Senior Data Engineer
New
J
JobgetherData engineering
Work remotely from eligible locations across Europe and Latin America.Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Work comfortably in an English-speaking professional environment
- Required Skills
- PostgreSQLPythonSQLGCPAirflowData modelingdbt
Requirements
- Proven experience building production data platforms from the ground up, including architecture, implementation, deployment, monitoring, and operations.
- Production experience with workflow orchestration tools such as Airflow, Dagster, Prefect, or Temporal; Airflow is preferred.
- Practical production experience with Google Cloud Platform and dbt.
- Experience developing asynchronous, event-driven systems using messaging platforms or task queues.
- Strong Python skills, including asynchronous HTTP requests and concurrent processing with asyncio, aiohttp, or HTTPX.
- Advanced SQL knowledge and practical PostgreSQL experience in data modeling, query optimization, incremental loading, and performance tuning.
- Experience with data contracts, raw and curated storage layers, retention policies, automated validation, and data quality monitoring.
- Hands-on experience building and maintaining production web-scraping pipelines with tools such as Scrapy, Playwright, Selenium, or Beautiful Soup.
- Ability to make technical decisions, manage complex projects, and own data infrastructure across the development lifecycle.
- Familiarity with matching and consolidating company and individual records across multiple sources is highly desirable.
- LLM-based structured data extraction or company or people data enrichment experience is an advantage.
- Familiarity with BigQuery and Dataform is a plus.
Responsibilities
- Own the architecture, deployment, optimization, and maintenance of a production-grade data platform.
- Build concurrent ingestion pipelines for provider APIs, webhooks, files, and web scraping, including rate limiting and error handling.
- Implement scheduled pipelines, historical backfills, dependency management, and event-driven processing.
- Create retries, dead-letter queues, replay mechanisms, and idempotent upserts for failure recovery.
- Preserve source payloads in Google Cloud Storage and transform data with dbt across raw, standardized, and curated layers.
- Define data contracts, validation rules, pipeline tests, retention policies, and monitoring systems.
- Design PostgreSQL schemas and optimize queries and incremental loading strategies.
- Integrate third-party enrichment providers and LLM-based structured extraction from text.
- Maintain lineage between source data and curated records for reprocessing, recovery, and auditing.
- Improve platform throughput, reliability, maintainability, and infrastructure efficiency.
View Full Description & ApplyYou'll be redirected to the employer's site
