Data Platform Engineer
New
O
OpsBrasil Serviços Cloud LTDAData engineering
100% remote; Open to candidate from all LATAMContract
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Professional written and spoken English.
- Required Skills
- PythonSQLSparkDatabricksAWS LambdaPySpark
Requirements
- Have production experience with Databricks workspaces, jobs, and workflows.
- Have strong Spark and PySpark experience, including tuning for real data volumes.
- Have production-grade Python experience.
- Have experience building or migrating Azure Data Factory pipelines.
- Understand the Azure Data Factory activity model.
- Have experience with AWS data services: S3, Glue, Athena, Lambda, and Step Functions.
- Have advanced SQL skills, including reading and reasoning about stored procedures.
- Have professional written and spoken English.
- Nice to have: Delta Lake, Unity Catalog, Azure Synapse, Terraform, Airflow/MWAA, dbt, Kafka, Databricks certification, data modeling, Great Expectations, SAS/analytics platform integration, or CRM data experience.
Responsibilities
- Convert Azure Data Factory pipelines into reusable Databricks workflows on AWS.
- Rehost Databricks workspaces onto AWS and migrate ADLS Gen2 storage to S3.
- Rewrite ADF Web Activities as Lambda functions or Step Functions tasks.
- Replace ADF-specific scaling with native Databricks mechanisms.
- Build and tune PySpark transformations for production data volumes.
- Replace Azure Synapse Serverless with Databricks SQL Warehouse.
- Reconcile migrated data against source systems.
View Full Description & ApplyYou'll be redirected to the employer's site
