Senior Data Engineer (Databricks) - Contractor

New
J
JobgetherData engineering
Fully remote contract opportunity available to professionals based in the United States.ContractSenior
Salary not disclosed
Apply NowOpens the employer's application page

Job Details

Experience
At least 2 years of hands-on experience designing and developing data pipelines using Databricks.
Required Skills
PythonSQLApache AirflowSparkdbtDatabricksPySpark

Requirements

  • Have at least 2 years of hands-on experience designing and developing data pipelines using Databricks.
  • Know Databricks features including Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
  • Have experience building YAML configuration-driven pipelines for batch, micro-batch, and streaming processing.
  • Have experience with data ingestion technologies such as Databricks Auto Loader.
  • Understand workflow orchestration and have experience with Databricks Jobs and Workflows, Apache Airflow, or dbt.
  • Have experience implementing data quality, pipeline auditing, monitoring, and observability solutions.
  • Be proficient with distributed processing technologies including Apache Spark, Presto, and Amazon EMR.
  • Have experience with cloud data warehouses such as Amazon Redshift, Snowflake, and Google BigQuery.
  • Have strong programming skills in SQL, stored procedures, and object-oriented languages such as Java and Python, including PySpark.
  • Be familiar with DevOps practices and CI/CD pipeline development, particularly Databricks Asset Bundles, automated testing, and validation.
  • Be able to work independently while collaborating with architects and cross-functional engineering teams.
  • Databricks Data Engineer Associate and/or Professional certification is preferred.
  • Experience delivering multiple Databricks implementation projects is preferred.
  • Experience developing reusable data engineering frameworks, accelerators, or enterprise-scale data platform solutions is preferred.
  • The initial contract engagement is six months, with the possibility of extension.

Responsibilities

  • Lead development and implementation of data engineering solutions on Databricks for batch and streaming workloads.
  • Design and develop solutions using Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
  • Build and optimize pipelines for batch, micro-batch, and streaming ingestion and processing.
  • Implement YAML configuration-driven pipelines and use ingestion technologies such as Auto Loader.
  • Design and maintain orchestration and workflow patterns using Databricks Jobs and Workflows, Apache Airflow, or dbt.
  • Implement data quality, pipeline auditing, monitoring, and observability solutions.
  • Collaborate with Solution Architects to turn architectural designs into technical components.
  • Break down development requirements and deliver assigned work within agreed timelines.
  • Develop reusable solution accelerators and contribute to Databricks Center of Excellence initiatives.
  • Explore emerging technologies and share technical knowledge and best practices with engineering colleagues.
View Full Description & ApplyYou'll be redirected to the employer's site
View details
Apply Now