Databricks Data Engineer
New
Remote CanadaContractSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 7+ years
- Required Skills
- PythonSQLETLGitSparkScalaDatabricksPySpark
Requirements
- Bachelor's degree in Computer Science, Software Engineering, or related field, or equivalent experience.
- 7+ years of professional data engineering experience.
- Hands-on experience with Databricks Workspaces, Clusters, Jobs, and Unity Catalog.
- Strong knowledge of Delta Lake, including ACID transactions, Time Travel, OPTIMIZE, and VACUUM.
- Proficiency in PySpark, Spark SQL, and Scala.
- Experience building and managing Databricks Workflows and ETL/ELT pipelines.
- Expertise in Medallion Architecture and incremental data loading patterns.
- Cloud platform experience (AWS or Azure).
- Experience with DevOps practices: Git, CI/CD, and unit/automated testing.
- Experience using AI-powered developer tools and understanding of responsible AI usage.
Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Build and optimize ETL/ELT workflows to ingest, transform, and curate large datasets.
- Implement data solutions following Medallion Architecture (Bronze, Silver, Gold).
- Develop high-performance PySpark and Spark SQL transformations.
- Optimize Delta Lake tables using partitioning, optimization, vacuuming, and time travel capabilities.
- Orchestrate and monitor Databricks Workflows and scheduled jobs.
- Collaborate with analytics, engineering, and product teams to deliver reliable data products.
- Apply software engineering best practices, including version control, automated testing, and CI/CD.
- Utilize AI-assisted development tools for code generation, debugging, documentation, and productivity.
View Full Description & ApplyYou'll be redirected to the employer's site
