Senior Platform Data Engineer
New
Australia, Canada, France, Germany, United Kingdom, United StatesFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- Minimum of 5 years
- Required Skills
- PythonSQLKafkaDatabricksPySpark
Requirements
- 5+ years of experience in data engineering.
- Expert-level Databricks skills including Delta Live Tables, PySpark, Unity Catalog, and Feature Store.
- Hands-on experience with real-time data ingestion using Kafka, Spark Structured Streaming, or similar frameworks.
- Strong proficiency in SQL and Python (pandas, PySpark) for data transformation and feature engineering.
- Experience administering Databricks workspaces including cluster policies, compute management, and access controls.
- Bachelor's degree in a related field.
- Familiarity with clinical data models and healthcare data sources is strongly preferred.
- Experience with Epic data extraction methods (SDE, FHIR, epic-ws) is a significant plus.
Responsibilities
- Stream clinical data from Epic SDE and ADT feeds into Databricks for AI model consumption.
- Curate shared clinical feature tables for model training and monitoring.
- Develop and manage RAG infrastructure, including document ingestion, chunking strategies, and embedding models.
- Administer vector databases and build retrieval pipelines including hybrid search and reranking.
- Establish data quality gates, monitor retrieval metrics, and perform Databricks workspace configuration and governance.
View Full Description & ApplyYou'll be redirected to the employer's site
