Senior Data Engineer

New
J
JobgetherData platform
Based in BrazilFull-TimeSenior
Salary80,000 - 120,000 USD per year
Apply NowOpens the employer's application page

Job Details

Experience
At least 5 years of experience building and operating production data platforms
Required Skills
PostgreSQLPythonSQLAirflowdbt

Requirements

  • Have at least 5 years of experience building and operating production data platforms, from architecture through deployment and maintenance.
  • Have production experience with Google Cloud Platform (GCP), dbt, and workflow orchestration tools such as Airflow, Dagster, Prefect, or Temporal.
  • Have strong Python programming skills, including asynchronous HTTP and concurrent processing with asyncio, aiohttp, or HTTPX.
  • Have advanced SQL and PostgreSQL skills, including data modeling, query optimization, and incremental loading strategies.
  • Have experience designing and operating event-driven systems, messaging or task queues, asynchronous processing, and failure recovery.
  • Be familiar with data contracts, raw and curated storage layers, retention policies, automated testing, and data quality monitoring.
  • Be able to design resilient ingestion and transformation pipelines for large data volumes and multiple source formats.
  • Be able to make sound architectural decisions independently.
  • Experience in a startup or small engineering team is an advantage.
  • Experience with web scraping, entity resolution, data enrichment, or LLM-based data extraction is a plus.

Responsibilities

  • Design and build a multi-source data platform for concurrent ingestion, event-driven processing, and orchestrated batch workflows.
  • Develop ingestion pipelines for APIs, webhooks, files, and web scraping, including rate-limit management and failure handling.
  • Orchestrate scheduled workflows and historical data backfills using Airflow, Dagster, Prefect, or similar tools.
  • Build messaging and task queue workflows with retries, dead-letter queues, and controlled replay capabilities.
  • Store original data payloads in cloud object storage and transform data with dbt into raw, standardized, and curated layers.
  • Implement idempotent upserts, data contracts, automated validation, pipeline testing, and monitoring for data freshness and failures.
  • Design and optimize PostgreSQL schemas, data models, and loading patterns.
  • Integrate data enrichment and LLM-based structured extraction into data workflows.
  • Own platform architecture, deployment, production operations, and ongoing improvements.
  • Identify bottlenecks and improve pipeline reliability, scalability, and maintainability.
View Full Description & ApplyYou'll be redirected to the employer's site
80,000 - 120,000 USD per year
Apply Now