Quality Engineer (Data)
New
C
Capital Technology GroupData Engineering
Remote (US)Full-TimeSenior
Salary$75k–$110k annually
Apply NowOpens the employer's application page
Job Details
- Experience
- 7+ years
- Required Skills
- AWSPythonSQLAgileApache AirflowSparkPySpark
Requirements
- Bachelor's degree in Computer Science, Information Systems, Engineering, or equivalent experience.
- 7+ years of professional experience in software quality assurance, testing, or quality engineering.
- Strong programming experience with Python and test frameworks (e.g., Pytest).
- Expertise in data quality principles including accuracy, completeness, and schema consistency.
- Experience with relational databases and SQL.
- Experience testing data pipelines, ETL/ELT processes, and large-scale transformations.
- Experience with distributed data processing technologies like Apache Spark or PySpark.
- Familiarity with Apache Airflow or similar workflow orchestration tools.
- Experience with cloud-based data platforms, preferably AWS.
- Experience supporting Agile software development teams.
- Strong analytical and problem-solving skills for investigating data discrepancies.
Responsibilities
- Develop and implement data quality strategies, standards, and maintenance processes for data pipelines and analytical datasets.
- Design and execute automated and manual tests for data accuracy, completeness, integrity, and consistency.
- Build automated quality checks and end-to-end tests using Python, PySpark, SQL, and Apache Airflow.
- Validate data transformations across Apache Spark, Python, and AWS services.
- Develop reusable testing frameworks and utilities for data pipelines and CI/CD.
- Investigate data anomalies, pipeline failures, and schema changes, partnering with Data Engineers on resolution.
- Monitor data quality metrics and support automated regression testing.
- Collaborate with stakeholders to translate regulatory requirements into data quality controls.
View Full Description & ApplyYou'll be redirected to the employer's site
