Senior Manager, Machine Learning (Data Operations)
C
CoalitionInsurance, Cybersecurity
Remote-firstFull-TimeManager
Salary134,400 - 200,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonSQLMachine LearningQuality Assurance
Requirements
- 5+ years in ML data operations, data labeling, or a related field (ML engineering, data science, or data engineering).
- Deep understanding of annotation quality frameworks: IAA, consensus labeling, gold standard evaluation, error taxonomy, and calibration workflows.
- Direct experience managing labeling platforms (Label Studio strongly preferred; Scale AI, Labelbox, Prodigy, or similar acceptable).
- Track record managing outsourced labeling vendors or BPOs for ML data production.
- Familiarity with common ML labeling tasks such as text classification, NER, document extraction, and intent detection.
- Proficiency in Python and SQL.
- Strong analytical skills with the ability to define and drive continuous improvement in labeling processes.
- Ability to push back on ambiguous task designs and poor quality standards.
Responsibilities
- Define annotation guidelines, taxonomies, and edge-case protocols for labeling programs.
- Establish gold standard datasets, inter-annotator agreement (IAA) targets, and audit sampling processes.
- Manage and maintain Label Studio configuration, workflows, and ML integration points.
- Partner with ML engineers and product managers to translate requirements into labeling tasks.
- Source, onboard, and manage external labeling vendors and BPOs to ensure quality and throughput SLAs.
- Define and track operational metrics including accuracy, IAA, cost per label, and turnaround time.
View Full Description & ApplyYou'll be redirected to the employer's site
