Senior Data Engineer
New
Our team works in a hybrid model from the San Francisco Bay Area. We will prioritize candidates who are able to work 2 days per week from our office, and we will consider highly qualified remote candidates who can travel quarterly to the San Francisco office.Full-TimeSenior
Salary190,000 - 220,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years building data-intensive SaaS platforms (L5: 8+ years with technical leadership)
- Required Skills
- AWSPythonSQLAirflowSparkTerraformData modelingDatabricksHIPAA
Requirements
- 5+ years building data-intensive SaaS platforms (L5: 8+ years with technical leadership).
- Deep, hands-on expertise with Spark and distributed data processing.
- Strong SQL and data modeling / warehouse design (dimensional modeling, Delta / Lakehouse).
- Proven track record scaling a product to an enterprise level.
- Experience with orchestration (Airflow), IaC (Terraform), and CI/CD for data.
- Experience with data-quality / testing frameworks such as dbt tests or Great Expectations.
- Ability to quickly understand complex modeling workflows and the business need driving them.
- Ships high-caliber, well-tested code with strong attention to detail.
- Experience with healthcare data (claims, eligibility) and handling PHI / PII under HIPAA.
- Thrives under minimal supervision in a rapidly changing, ambiguous start-up environment.
Responsibilities
- Scale Arbital's healthcare data pipelines and lakehouse on AWS and Databricks, and own the underlying architecture.
- Implement and scale actuarially sound healthcare financial calculations in Spark.
- Build and maintain orchestration (Airflow) and CI/CD so enrichment and aggregation workflows are reliable, observable, and reproducible.
- Own data quality, integrity, privacy, security, and HIPAA compliance through automated testing and quality-control procedures.
- Collaborate with actuarial and delivery teams that primarily work in Python and R.
- Partner with data scientists to deploy and monitor machine learning models in production.
- Lead technical design reviews and contribute to platform-wide architecture decisions.
- Establish data observability, lineage, and SLAs, and tune Spark/Databricks jobs for performance and cost.
- Raise the engineering bar through code review, mentorship, and setting data-engineering standards across the team.
View Full Description & ApplyYou'll be redirected to the employer's site
