Senior Engineer – Data Engineering (Databricks) - Contractor

New
J
JobgetherData engineering
Fully remote working arrangement within India.ContractSenior
Salary not disclosed
Apply NowOpens the employer's application page

Job Details

Experience
At least 5 years of professional experience in data engineering, including a minimum of 2 years of hands-on experience designing and developing data pipelines using Databricks.
Required Skills
PythonSQLApache AirflowSparkDatabricksPySpark

Requirements

  • Have at least 5 years of professional data engineering experience, including at least 2 years designing and developing Databricks data pipelines.
  • Demonstrate practical expertise with Delta Lake, Delta Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
  • Have experience designing scalable YAML configuration-driven pipelines for batch, micro-batch, and streaming processing.
  • Understand data ingestion patterns and have practical experience with Databricks Auto Loader.
  • Have experience with Databricks Jobs and Workflows, Apache Airflow, dbt, or comparable orchestration tools.
  • Know data quality management, pipeline auditing, monitoring, and observability frameworks.
  • Have experience with big data technologies, particularly Apache Spark, Presto, and Amazon EMR.
  • Have advanced SQL skills, including stored procedures, query optimization, and complex transformations.
  • Have strong programming skills in Python, PySpark, Java, or another object-oriented language.
  • Have experience implementing DevOps and CI/CD pipelines, including automated testing and deployment validation.
  • Have delivered multiple Databricks-based data engineering projects in enterprise environments.

Responsibilities

  • Design and implement scalable data engineering solutions using Databricks and cloud-based lakehouse architectures.
  • Build and maintain pipelines using Delta Lake, Delta Live Tables, Unity Catalog, and Delta Uniform.
  • Develop YAML configuration-driven batch, micro-batch, and streaming pipelines.
  • Implement ingestion and transformation workflows using Databricks Auto Loader and Apache Spark.
  • Develop workflow orchestration using Databricks Jobs and Workflows, Apache Airflow, or dbt.
  • Collaborate with solution architects on enterprise data platform architecture and client requirements.
  • Implement data quality, auditing, monitoring, and observability frameworks.
  • Optimize data processing performance, reliability, scalability, and resource utilization.
  • Implement CI/CD processes using Databricks Asset Bundles, including automated testing and deployment validation.
  • Contribute reusable frameworks, solution accelerators, technical standards, and engineering best practices.
View Full Description & ApplyYou'll be redirected to the employer's site
View details
Apply Now