Senior Data Engineer (Databricks)
O
Object EdgeData engineering
Remote - United StatesContractSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 2 years of hands-on data pipeline design and development experience with Databricks.
- Required Skills
- PythonSQLAirflowSparkDatabricksPySpark
Requirements
- Have at least 2 years of hands-on experience designing and developing data pipelines with Databricks.
- Have experience with Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
- Design and build YAML configuration-driven data pipelines for batch, micro-batch, and streaming ingestion and processing.
- Have experience with Auto Loader and modern data engineering practices.
- Have experience building job and workflow orchestration patterns with Databricks jobs/workflows or tools such as Airflow or dbt.
- Have exposure to data quality, pipeline audit, and observability solutions using Databricks features or tools such as Great Expectations, Collibra, or dbt.
- Have proficiency with Apache Spark, Presto, or Amazon EMR.
- Have experience with Amazon Redshift, Snowflake, or Google BigQuery.
- Have strong SQL, stored procedures, and object-oriented programming skills using languages such as Java, Python, or PySpark.
- Be familiar with DevOps and CI/CD pipelines using Databricks Asset Bundles, automated validation, and testing.
- Databricks Data Engineering Associate or Professional certification and multiple Databricks project deliveries are ideal.
Responsibilities
- Develop and implement data solutions using Databricks features including Delta Lake, Uniform (Iceberg), Delta Live Tables, and Unity Catalog.
- Collaborate with cross-functional teams to design and optimize batch and streaming data pipelines.
- Ensure data pipeline quality and efficiency.
- Help design and implement components of client data platform solutions under the direction of a Solution Architect.
- Break down complex development tasks and complete development items on time.
- Contribute to Databricks Center of Excellence initiatives and develop Databricks solution accelerators.
- Stay current with emerging technologies and Databricks platform features.
View Full Description & ApplyYou'll be redirected to the employer's site
