Senior Databricks Migration Engineer
J
JobgetherData engineering
Fully remote opportunity available to candidates throughout the continental United States., Preference for candidates in the U.S. East Coast time zone.Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years of experience in data engineering, data system development, or related roles; 5+ years of experience working with cloud platforms; at least 1 year of experience leading complex, cross-functional data projects and technical teams.
- Required Skills
- PythonSQLMicrosoft Power BISparkPySpark
Requirements
- Hold a bachelor’s degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field.
- Have 5+ years of experience in data engineering, data system development, or related roles.
- Have 5+ years of experience working with cloud platforms such as Azure, AWS, or GCP.
- Have at least 1 year of experience leading complex, cross-functional data projects and technical teams.
- Bring strong expertise in data engineering principles, data modeling, ETL processes, and data pipeline development.
- Have hands-on experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, cloud storage, and distributed computing platforms.
- Be proficient in SQL and Python/PySpark for data manipulation and pipeline development.
- Have experience with Azure Data Lake storage and processing services.
- Have experience designing, building, and optimizing pipelines for ingestion, transformation, and loading.
- Have experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, metadata-driven ingestion, and PySpark data quality frameworks.
- Understand query performance optimization, scalability, and efficient data modeling.
Responsibilities
- Lead migration of legacy SQL Server stored procedures and Azure Data Factory pipelines to Databricks Lakehouse and Delta Lake.
- Translate relational data warehouse architectures into Bronze, Silver, and Gold Lakehouse frameworks.
- Design reusable ETL/ELT frameworks using PySpark, Delta Live Tables, and Databricks Workflows.
- Architect and optimize Gold Layer dimensional models and star schemas for Power BI performance.
- Optimize Databricks SQL Warehouses for Power BI DirectQuery and Import workloads.
- Apply Delta Lake optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views.
- Define compute, autoscaling, partitioning, and file-size standards, and monitor DBU consumption for cost optimization.
- Design data security and governance using Unity Catalog, including row- and column-level security and alignment with Microsoft Entra ID and enterprise RBAC.
- Lead technical workshops, pair programming, and code reviews, and produce architecture documentation and optimization playbooks.
- Establish knowledge-transfer practices so internal teams can operate and enhance the platform independently.
View Full Description & ApplyYou'll be redirected to the employer's site
