Senior Data Engineer
New
J
JobgetherData engineering
Based in BrazilFull-TimeSenior
SalaryAnnual gross base salary ranging from $50,000 to $80,000 USD
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 7 years of professional experience in data engineering, including a minimum of 3 years operating production-grade streaming data pipelines.
- Required Skills
- PythonSQLApache AirflowKafkaCI/CD
Requirements
- At least 7 years of professional data engineering experience, including at least 3 years operating production-grade streaming data pipelines.
- Hands-on Apache Spark experience, particularly Structured Streaming, state stores such as RocksDB, checkpointing, stateful processing, and performance optimization.
- Production experience with lakehouse table formats, ideally Apache Iceberg, including partitioning, table maintenance, compaction, and snapshot management; Delta Lake or Apache Hudi experience is relevant.
- Experience operating Spark workloads on AWS EMR, Amazon EKS, or comparable platforms, including cluster management, capacity planning, autoscaling, resource allocation, and cost optimization.
- Understanding of CDC principles and Kafka semantics, including offsets, consumer configurations, message delivery guarantees, and event processing reliability.
- Strong proficiency in Python and SQL; Scala or Java experience is an advantage.
- Experience with Apache Airflow or similar workflow orchestration tools, CI/CD pipelines, and infrastructure-as-code practices.
- Systematic debugging skills, including interpreting logs, investigating complex issues, formulating hypotheses, and validating solutions with data.
- Ability to work independently and collaborate with distributed engineering teams and cross-functional stakeholders.
- Snowflake, Grafana, Loki, or Alloy experience is a plus.
- Experience building self-service data platforms or enabling engineering teams to manage data workflows is an advantage.
- Familiarity with Kubernetes is a plus.
Responsibilities
- Design, build, and maintain Apache Spark Structured Streaming applications that process CDC events into near-real-time data models.
- Implement and optimize stateful processing, slowly changing dimensions, checkpointing, state stores, and recovery mechanisms.
- Manage and scale AWS EMR clusters, including resource allocation, managed scaling, memory optimization, and workload consolidation.
- Maintain Apache Iceberg tables through partitioning, compaction, snapshot expiration, and delete-file cleanup.
- Onboard business domains by configuring CDC sources and destinations through Estuary Flow, setting up Kafka-compatible topics, and validating backfills in Snowflake.
- Investigate and resolve production incidents across the data platform, focusing on root causes.
- Develop monitoring and diagnostic tools, including Spark log integration with Grafana Loki and watchdog mechanisms.
- Improve deployment workflows, configuration management, automated pipelines, and operational safeguards.
- Support partner engineering teams with data ingestion, backfill requests, technical escalations, and platform guidance.
View Full Description & ApplyYou'll be redirected to the employer's site
