Senior Manager, Machine Learning (Data Operations)
New
C
CoalitionCybersecurity Insurance
USAFull-TimeManager
Salary134,400 - 200,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonSQLMachine LearningQuality Assurance
Requirements
- 5+ years in ML data operations, data labeling, or a related field such as ML engineering, data science, or data engineering.
- Deep understanding of annotation quality frameworks (IAA, consensus labeling, gold standard evaluation, error taxonomy, calibration workflows).
- Direct experience managing labeling platforms (Label Studio strongly preferred; Scale AI, Labelbox, or Prodigy acceptable).
- Track record managing outsourced labeling vendors or BPOs for ML data production.
- Familiarity with ML labeling tasks like text classification, NER, document extraction, and intent detection.
- Proficiency in Python and SQL.
- Ability to push back on ambiguous task designs and poor quality standards.
Responsibilities
- Define annotation guidelines, taxonomies, and edge-case protocols for labeling programs.
- Establish gold standard datasets, IAA targets, and audit sampling processes.
- Manage and configure the Label Studio platform, defining project workflows and pipeline requirements.
- Source, onboard, and manage external labeling vendors and BPOs.
- Set quality SLAs, run calibration sessions, and manage feedback loops to vendors.
- Track operational metrics such as label accuracy, IAA scores, and cost per label to drive continuous improvement.
View Full Description & ApplyYou'll be redirected to the employer's site
