Senior Data Engineer
New
C
CI&TTechnology Services
BrazilFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- DynamoDBETLSparkDatabricks
Requirements
- Proficiency in Databricks (central requirement).
- Expertise in Apache Spark for pipeline execution.
- Strong background in Data Engineering and ETL processes.
- Mastery of Medallion architecture (Bronze, Silver, Gold layers).
- Proven experience with data pipeline orchestration.
- Experience implementing data quality frameworks in production environments.
- Familiarity with AWS cloud environments (specifically S3 and DynamoDB).
- Ability to work autonomously with minimal ramp-up time.
- Experience operating in fast-paced 'build-first, design-later' environments.
- Capacity for cross-squad collaboration.
Responsibilities
- Execute data lifecycle tasks including ingestion into S3, Medallion architecture (Bronze/Silver/Gold) processing, feature store integration, and distribution.
- Replace manual, sequential pipeline orchestration with automated native Databricks orchestration.
- Implement end-to-end data quality processes across all pipeline layers.
- Develop and parameterize existing flows based on predefined architecture.
- Build a billing engine for data product consumption.
- Develop an archival engine to standardize data distribution across channels like SFTP, buckets, and APIs.
View Full Description & ApplyYou'll be redirected to the employer's site
