Databricks Architect
United StatesFull-TimeMiddle
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 12+ years
- Required Skills
- AWSHadoopDatabricksAWS LambdaPySpark
Requirements
- 12+ years of experience in Architecture, design and data engineering development of large-scale data ecosystems
- Databricks Design and Architecture: Design and architect scalable and secure Databricks solutions platform, optimizing performance, cost, and reliability
- Hands on Databricks/Dataops Architect that can help reduce consumption, and improve performance issues
- Ability to build monitoring tools
- Good exposure to AWS
- Strong communication and leadership skills
- Self-driven and work independently
- Experience in Performance initiatives and drive recommendations (identify tuning opportunities, creation of performance environment etc.)
- Experience with hadoop environment
- Databricks platform knowledge
- Unity Catalog migration implementation exposure
- Spark Streaming pipeline design exposure
Responsibilities
- Design and implement scalable, end-to-end data solutions for ETL, real-time streaming, and machine learning workloads on the Databricks platform
- Provide technical leadership and strategic guidance on Databricks adoption, including developing reference architectures and best practices
- Ensure optimal performance by tuning Spark jobs, optimizing data pipelines, and resolving complex technical issues
- Architect, develop, and debug PySpark jobs and data models, integrating Databricks with cloud services such as AWS S3, EMR, and Lambda
- Implement and enforce data governance, security, and access control policies (e.g., encryption, auditing) on the Databricks environment
View Full Description & ApplyYou'll be redirected to the employer's site
