Senior Data Engineer
New
J
JobgetherData platform
Based in BrazilFull-TimeSenior
Salary80,000 - 120,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 5 years of experience building and operating production data platforms
- Required Skills
- PostgreSQLPythonSQLAirflowdbt
Requirements
- Have at least 5 years of experience building and operating production data platforms, from architecture through deployment and maintenance.
- Have production experience with Google Cloud Platform (GCP), dbt, and workflow orchestration tools such as Airflow, Dagster, Prefect, or Temporal.
- Have strong Python programming skills, including asynchronous HTTP and concurrent processing with asyncio, aiohttp, or HTTPX.
- Have advanced SQL and PostgreSQL skills, including data modeling, query optimization, and incremental loading strategies.
- Have experience designing and operating event-driven systems, messaging or task queues, asynchronous processing, and failure recovery.
- Be familiar with data contracts, raw and curated storage layers, retention policies, automated testing, and data quality monitoring.
- Be able to design resilient ingestion and transformation pipelines for large data volumes and multiple source formats.
- Be able to make sound architectural decisions independently.
- Experience in a startup or small engineering team is an advantage.
- Experience with web scraping, entity resolution, data enrichment, or LLM-based data extraction is a plus.
Responsibilities
- Design and build a multi-source data platform for concurrent ingestion, event-driven processing, and orchestrated batch workflows.
- Develop ingestion pipelines for APIs, webhooks, files, and web scraping, including rate-limit management and failure handling.
- Orchestrate scheduled workflows and historical data backfills using Airflow, Dagster, Prefect, or similar tools.
- Build messaging and task queue workflows with retries, dead-letter queues, and controlled replay capabilities.
- Store original data payloads in cloud object storage and transform data with dbt into raw, standardized, and curated layers.
- Implement idempotent upserts, data contracts, automated validation, pipeline testing, and monitoring for data freshness and failures.
- Design and optimize PostgreSQL schemas, data models, and loading patterns.
- Integrate data enrichment and LLM-based structured extraction into data workflows.
- Own platform architecture, deployment, production operations, and ongoing improvements.
- Identify bottlenecks and improve pipeline reliability, scalability, and maintainability.
View Full Description & ApplyYou'll be redirected to the employer's site