Senior Data Platform Engineer II (Databricks)
Remote, United StatesFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 6+ years experience as an engineer building and optimizing highly scalable distributed data systems (e.g., Databricks, Spark, or Snowflake).
- Required Skills
- AWSDockerSQLETLKubernetesSnowflakeSparkCI/CDTerraformData modelingDatabricksEHRHIPAADistributed Systems
Requirements
- BS/BTech (or higher) in Computer Science, Engineering or a related field or equivalent experience.
- 6+ years experience as an engineer building and optimizing highly scalable distributed data systems (e.g., Databricks, Spark, or Snowflake).
- 3+ years of experience working with SQL and data modeling on large multi-table data sets.
- 3+ years of experience acting as a trusted technical decision-maker in a team setting, solving for short-term and long-term business value.
- 3+ years of experience coaching other engineers.
- Deep expertise in managing Databricks workspaces, including Unity Catalog for data governance, lineage, and fine-grained access control.
- Advanced proficiency with Terraform (or similar) to automate the provisioning and scaling of Databricks clusters, cloud resources (AWS preferred), and networking.
- In-depth knowledge of distributed systems, including partitioning, liquid clustering/Z-Ordering, sharding, and high-availability strategies for petabyte-scale data.
- Proven track record in performance monitoring and query tuning for distributed workloads to ensure system reliability and cost-efficiency.
- Experience designing and optimizing high-throughput ETL/ELT pipelines and ingestion systems (batch and streaming) using Spark.
- Experience building robust CI/CD pipelines for data infrastructure and deploying services using containerization (Docker, Kubernetes).
- Expertise in building systems that handle protected information, with specific experience in HIPAA and SOX compliance frameworks.
- Experience navigating health-tech data complexities, such as Electronic Health Records (EHR), clinical data formats (HL7/FHIR), and claims data.
Responsibilities
- Architect and manage the high-performance, distributed data environments that power our healthcare analytics.
- Develop and implement scalable and performant solutions.
- Partner, as a peer, with Engineering Managers, Product Managers, and stakeholders throughout Aledade to develop and execute technical roadmaps using Agile processes.
- Mentor and coach more junior engineers including thorough pull request reviews for other developers and be receptive to critical feedback on your own work.
View Full Description & ApplyYou'll be redirected to the employer's site
