Senior Spark QA Engineer
New
IndiaFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonETLJavaKubernetesSparkCI/CDDatabricks
Requirements
- 5+ years of QA or software testing experience with strong hands-on expertise in Apache Spark and distributed data systems.
- Strong understanding of Spark architecture, optimization techniques, and distributed computing principles.
- Experience in functional, integration, and performance testing of large-scale data processing systems.
- Proficiency in Python or Java for test automation and framework development.
- Experience working with Kubernetes and cloud-based Spark environments (e.g., EMR, Dataproc, Databricks).
- Familiarity with CI/CD pipelines and automation frameworks for testing and deployment validation.
- Strong analytical and debugging skills with the ability to diagnose complex system-level issues.
Responsibilities
- Perform functional, integration, and automated testing of Apache Spark jobs, Spark SQL queries, and end-to-end ETL data pipelines across distributed systems.
- Execute performance, scalability, and benchmarking tests to evaluate Spark workloads under varying data volumes and cluster configurations.
- Set up, configure, and validate Spark environments across platforms such as YARN, Kubernetes, Databricks, EMR, Dataproc, Mesos, and standalone clusters.
- Identify performance bottlenecks, troubleshoot distributed system issues, and work closely with engineering teams to drive optimizations.
- Develop and maintain QA strategies, automation frameworks, and CI/CD-integrated testing processes for Spark-based applications.
- Lead QA initiatives, provide technical guidance, and mentor team members to improve testing quality and engineering practices.
View Full Description & ApplyYou'll be redirected to the employer's site
