Senior Manager, Machine Learning (Data Operations)
New
C
CoalitionCybersecurity Insurance
USAFull-TimeManager
Salary134,400 - 200,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonSQLMachine LearningQuality AssuranceNLP
Requirements
- 5+ years of experience in ML data operations, data labeling, or a related field such as ML engineering or data science.
- Deep understanding of annotation quality frameworks including IAA, consensus labeling, and gold standard evaluation.
- Direct experience managing labeling platforms, with Label Studio strongly preferred.
- Track record managing outsourced labeling vendors or BPOs for machine learning data production.
- Familiarity with common ML labeling tasks like text classification, NER, document extraction, and intent detection.
- Proficiency working in Python and SQL.
- Willingness to define and enforce high standards for data quality.
- Strong opinion on what constitutes high-quality versus low-quality data.
Responsibilities
- Define annotation guidelines, taxonomies, and edge-case protocols for each labeling program.
- Establish gold standard datasets, inter-annotator agreement (IAA) targets, and audit sampling processes.
- Serve as the primary user and requirements driver for the Label Studio labeling platform.
- Partner with ML engineers, data scientists, and product managers to translate requirements into labeling tasks.
- Source, onboard, and manage external labeling vendors and BPOs to set quality SLAs and run calibration sessions.
- Track operational metrics like label accuracy, cost per label, and turnaround time to drive continuous improvement.
View Full Description & ApplyYou'll be redirected to the employer's site
