Senior Data Engineer
New
C
CleraData infrastructure
Remote within the United States.Full-TimeSenior
Salary120,000 - 160,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 5 years building and operating production data platforms; 8 or more years is a plus.
- Required Skills
- PostgreSQLPythonSQLGCPAirflowdbt
Requirements
- Have at least 5 years building and operating production data platforms; 8 or more years is a plus.
- Have owned data platforms from architecture through deployment and ongoing support.
- Have production experience with workflow orchestration tools such as Airflow, Dagster, Prefect, or Temporal, including dependencies, retries, and recovery.
- Have hands-on production experience with dbt and Google Cloud Platform.
- Have strong Python skills, including asynchronous HTTP and concurrent processing with tools such as asyncio, aiohttp, or HTTPX.
- Have strong SQL and PostgreSQL experience in data modeling, query optimization, and incremental loading.
- Have experience building event-driven systems and implementing data contracts, retention policies, validation, testing, and monitoring.
- Web scraping, entity resolution, data enrichment, and LLM-based extraction experience are helpful.
- BigQuery and Dataform familiarity is a plus.
Responsibilities
- Design and build a multi-source platform for concurrent ingestion, event-driven processing, and orchestrated batch workflows.
- Develop resilient intake from provider APIs, webhooks, files, and web scraping, including rate limits and failure handling.
- Orchestrate scheduled workflows and historical backfills with tools such as Airflow, Dagster, or Prefect.
- Build event-driven processing with messaging or task queues, including retries, dead-letter queues, and controlled replay.
- Store original payloads in Google Cloud Storage and use dbt to transform data into raw, standardized, and curated layers.
- Implement idempotent upserts, data contracts, automated validation, pipeline tests, and monitoring for freshness and failures.
- Design and optimize PostgreSQL schemas and loading patterns for downstream use.
- Integrate provider enrichment and LLM-based structured extraction into the platform.
View Full Description & ApplyYou'll be redirected to the employer's site
