Senior Engineer – Data Engineering (Databricks) - Contractor
New
J
JobgetherData engineering
Fully remote working arrangement within India.ContractSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 5 years of professional experience in data engineering, including a minimum of 2 years of hands-on experience designing and developing data pipelines using Databricks.
- Required Skills
- PythonSQLApache AirflowSparkDatabricksPySpark
Requirements
- Have at least 5 years of professional data engineering experience, including at least 2 years designing and developing Databricks data pipelines.
- Demonstrate practical expertise with Delta Lake, Delta Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
- Have experience designing scalable YAML configuration-driven pipelines for batch, micro-batch, and streaming processing.
- Understand data ingestion patterns and have practical experience with Databricks Auto Loader.
- Have experience with Databricks Jobs and Workflows, Apache Airflow, dbt, or comparable orchestration tools.
- Know data quality management, pipeline auditing, monitoring, and observability frameworks.
- Have experience with big data technologies, particularly Apache Spark, Presto, and Amazon EMR.
- Have advanced SQL skills, including stored procedures, query optimization, and complex transformations.
- Have strong programming skills in Python, PySpark, Java, or another object-oriented language.
- Have experience implementing DevOps and CI/CD pipelines, including automated testing and deployment validation.
- Have delivered multiple Databricks-based data engineering projects in enterprise environments.
Responsibilities
- Design and implement scalable data engineering solutions using Databricks and cloud-based lakehouse architectures.
- Build and maintain pipelines using Delta Lake, Delta Live Tables, Unity Catalog, and Delta Uniform.
- Develop YAML configuration-driven batch, micro-batch, and streaming pipelines.
- Implement ingestion and transformation workflows using Databricks Auto Loader and Apache Spark.
- Develop workflow orchestration using Databricks Jobs and Workflows, Apache Airflow, or dbt.
- Collaborate with solution architects on enterprise data platform architecture and client requirements.
- Implement data quality, auditing, monitoring, and observability frameworks.
- Optimize data processing performance, reliability, scalability, and resource utilization.
- Implement CI/CD processes using Databricks Asset Bundles, including automated testing and deployment validation.
- Contribute reusable frameworks, solution accelerators, technical standards, and engineering best practices.
View Full Description & ApplyYou'll be redirected to the employer's site