Senior Data Engineer (Databricks) - Contractor
New
J
JobgetherData engineering
Fully remote contract opportunity available to professionals based in the United States.ContractSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 2 years of hands-on experience designing and developing data pipelines using Databricks.
- Required Skills
- PythonSQLApache AirflowSparkdbtDatabricksPySpark
Requirements
- Have at least 2 years of hands-on experience designing and developing data pipelines using Databricks.
- Know Databricks features including Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
- Have experience building YAML configuration-driven pipelines for batch, micro-batch, and streaming processing.
- Have experience with data ingestion technologies such as Databricks Auto Loader.
- Understand workflow orchestration and have experience with Databricks Jobs and Workflows, Apache Airflow, or dbt.
- Have experience implementing data quality, pipeline auditing, monitoring, and observability solutions.
- Be proficient with distributed processing technologies including Apache Spark, Presto, and Amazon EMR.
- Have experience with cloud data warehouses such as Amazon Redshift, Snowflake, and Google BigQuery.
- Have strong programming skills in SQL, stored procedures, and object-oriented languages such as Java and Python, including PySpark.
- Be familiar with DevOps practices and CI/CD pipeline development, particularly Databricks Asset Bundles, automated testing, and validation.
- Be able to work independently while collaborating with architects and cross-functional engineering teams.
- Databricks Data Engineer Associate and/or Professional certification is preferred.
- Experience delivering multiple Databricks implementation projects is preferred.
- Experience developing reusable data engineering frameworks, accelerators, or enterprise-scale data platform solutions is preferred.
- The initial contract engagement is six months, with the possibility of extension.
Responsibilities
- Lead development and implementation of data engineering solutions on Databricks for batch and streaming workloads.
- Design and develop solutions using Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
- Build and optimize pipelines for batch, micro-batch, and streaming ingestion and processing.
- Implement YAML configuration-driven pipelines and use ingestion technologies such as Auto Loader.
- Design and maintain orchestration and workflow patterns using Databricks Jobs and Workflows, Apache Airflow, or dbt.
- Implement data quality, pipeline auditing, monitoring, and observability solutions.
- Collaborate with Solution Architects to turn architectural designs into technical components.
- Break down development requirements and deliver assigned work within agreed timelines.
- Develop reusable solution accelerators and contribute to Databricks Center of Excellence initiatives.
- Explore emerging technologies and share technical knowledge and best practices with engineering colleagues.
View Full Description & ApplyYou'll be redirected to the employer's site