Databricks Solution Architect
New
Fully remote work flexibility within the United States.Full-TimeSenior
Salary102,000 - 133,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 8+ years of experience in data engineering, including at least 4+ years working extensively with Databricks in production environments.
- Required Skills
- PythonSQLSparkCI/CDTerraformData modelingDatabricksPySpark
Requirements
- 8+ years of experience in data engineering, including 4+ years working extensively with Databricks in production environments.
- Deep expertise in Apache Spark (PySpark and Spark SQL), including performance tuning and distributed processing.
- Strong hands-on experience with Databricks ecosystem tools such as Delta Lake, Unity Catalog, Delta Live Tables, and Databricks Workflows.
- Proven experience deploying cloud-based data solutions on AWS, Azure, or GCP.
- Proficiency in Python and SQL; Scala experience is a plus.
- Strong understanding of medallion architecture and dimensional data modeling.
- Experience implementing CI/CD pipelines and DevOps practices using Git, Terraform, and data deployment frameworks.
- Demonstrated ability to lead technical initiatives end-to-end.
- Excellent communication skills.
- Strong problem-solving mindset.
Responsibilities
- Architect and lead the design and implementation of an enterprise Databricks-based lakehouse platform using technologies such as Delta Lake, Unity Catalog, Photon, and Databricks Workflows.
- Design and build scalable batch and streaming data pipelines using PySpark, Spark SQL, Structured Streaming, and Delta Live Tables.
- Define and enforce data platform standards including medallion architecture, CI/CD practices, testing frameworks, and observability.
- Lead data governance implementation using Unity Catalog, including access control, lineage tracking, and secure handling of sensitive data.
- Optimize Spark workloads for performance and cost efficiency through tuning strategies.
- Partner with data science and ML teams to operationalize machine learning models using MLflow and feature stores.
- Own cloud infrastructure architecture including networking, IAM, storage, and Infrastructure-as-Code using Terraform.
- Mentor and guide data engineering teams through architecture and code reviews.
- Collaborate with stakeholders to translate requirements into a scalable data platform roadmap.
- Ensure compliance with information security and data protection standards.
View Full Description & ApplyYou'll be redirected to the employer's site
