Senior Data Engineer
New
M
MachinifyHealthcare intelligence
Work from anywhere in the US!Full-TimeSenior
Salary180,000 - 220,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 6+ years
- Required Skills
- AWSPythonApache AirflowETLKafkaData engineeringData modeling
Requirements
- 6+ years of experience as a Data Engineer building production-grade pipelines.
- Strong expertise in Python, Spark SQL, and Airflow.
- Experience processing large-scale file-based datasets (CSV, Parquet, JSON) in production.
- Experience mapping and standardizing raw external data into canonical models.
- Familiarity with AWS or other cloud providers.
- Experience onboarding new customers and integrating non-standard external data.
- Ability to work across teams and own complex data workflows.
- Strong written and verbal communication skills.
- Experience with large-scale or messy datasets (healthcare, financial, logs).
- Experience building or willingness to learn streaming pipelines (Kafka, SQS).
Responsibilities
- Design and implement robust, production-grade pipelines using Python, Spark SQL, and Airflow.
- Lead efforts to canonicalize raw healthcare data (837 claims, EHR, partner data) into internal models.
- Own the full lifecycle of core pipelines from file ingestion to validated, queryable datasets.
- Onboard new customers by integrating their raw data into internal pipelines.
- Build resilient, idempotent transformation logic with data quality checks and observability.
- Refactor and scale existing pipelines to meet growing business needs.
- Tune Spark jobs and optimize distributed processing performance.
- Collaborate with Data Analysts, Scientists, Product Managers, and SMEs.
- Monitor pipeline health and participate in on-call rotations.
- Develop and champion internal best practices for pipeline development and data modeling.
View Full Description & ApplyYou'll be redirected to the employer's site
