Data Quality Engineer
New
P
Prolific AI Data Services
Location: North AmericaFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonSQL
Requirements
- 5+ years of experience in quality engineering, data or annotation quality, analytics engineering, trust and integrity, or ML/LLM evaluation operations.
- Strong proficiency in Python and SQL.
- Knowledge of statistical concepts such as sampling strategies, confidence levels, and agreement metrics.
- Track record of turning ambiguous quality problems into clear metrics and automated checks.
- Strong quality systems thinking.
- Hands-on experience instrumenting workflows and implementing automation.
- Demonstrated ability to influence cross-functional teams.
- Customer empathy regarding research and AI training use cases.
Responsibilities
- Own end-to-end quality design for Prolific managed service studies, including rubrics, acceptance criteria, defect taxonomies, severity models, and clear definitions of done.
- Define, implement, and maintain quality measurement systems, including sampling plans, golden sets, calibration protocols, agreement targets, adjudication workflows, and drift detection.
- Build and deploy automated quality checks and launch gates using Python and SQL.
- Design and run launch readiness processes, including pre-launch checks, pilot calibration, and rollback mechanisms.
- Partner with Product and Engineering to embed in-study quality controls and authenticity checks.
- Investigate quality and integrity issues, running root-cause analysis and driving corrective and preventive actions.
- Build dashboards and operating cadences to track defect rates, throughput versus quality trade-offs, and SLA adherence.
- Lead calibration sessions and coach QA leads and reviewers.
View Full Description & ApplyYou'll be redirected to the employer's site
