Senior Data Engineer (Databricks)
New
O
Object EdgeData engineering
Remote - BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 2 years of hands-on data pipeline design and development experience with Databricks
- Required Skills
- PythonSQLAirflowSparkdbtDatabricksPySpark
Requirements
- Have at least 2 years of hands-on experience designing and developing data pipelines with Databricks.
- Have experience with Delta Lake, Uniform (Iceberg), Delta Live Tables (Lakeflow Declarative Pipelines), and Unity Catalog.
- Design and build scalable, YAML configuration-driven data pipelines for batch, micro-batch, and streaming ingestion and processing.
- Have experience with Auto Loader and data pipeline orchestration using Databricks jobs/workflows or tools such as Airflow or dbt.
- Have exposure to data quality, pipeline audit, and observability solutions, including Databricks features or tools such as Great Expectations, Collibra, or dbt.
- Be proficient with Apache Spark, Presto, or Amazon EMR.
- Have experience with Amazon Redshift, Snowflake, or Google BigQuery.
- Have strong SQL and stored procedure skills, plus object-oriented programming skills in languages such as Java, Python, or PySpark.
- Be familiar with DevOps and CI/CD pipelines using Databricks Asset Bundles with automated validation and testing.
- Familiarity with NoSQL databases and container management systems is requested.
- Exposure to Terraform, AI/ML tools such as MLflow, prompt engineering, and data and AI agentic workflows is a plus.
- Databricks Data Engineering Associate and/or Professional certification and multiple Databricks project deliveries are ideal.
Responsibilities
- Develop and implement data solutions using Databricks, including Delta Lake, Uniform (Iceberg), Delta Live Tables, and Unity Catalog.
- Collaborate with cross-functional teams to design and optimize batch and streaming data pipelines.
- Ensure data quality and pipeline efficiency.
- Work under a Solution Architect to design and implement components of client data platform solutions.
- Break down complex development tasks and complete development items on time.
- Contribute to Databricks Center of Excellence initiatives and develop Databricks solution accelerators.
- Keep current with Databricks platform features and relevant technologies.
View Full Description & ApplyYou'll be redirected to the employer's site
