Senior Manager, Machine Learning (Data Operations)
New
C
CoalitionCyber Insurance
CanadaFull-TimeManager
Salary123,100 - 186,800 CAD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonSQLMachine LearningQuality Assurance
Requirements
- 5+ years in ML data operations, data labeling, or a related field (ML engineering, data science, or data engineering with labeling exposure).
- Deep understanding of annotation quality frameworks: IAA, consensus labeling, gold standard evaluation, and calibration workflows.
- Direct experience managing labeling platforms (Label Studio strongly preferred; Scale AI, Labelbox, or Prodigy acceptable).
- Track record managing outsourced labeling vendors or BPOs for ML data production.
- Familiarity with common ML labeling tasks such as text classification, NER, document extraction, and intent detection.
- Proficiency in Python and SQL.
- Ability to build tooling around labeling workflows or quality measurement.
- Strong opinion on labeled data quality and willingness to challenge ambiguous task designs.
Responsibilities
- Define annotation guidelines, taxonomies, and edge-case protocols for each labeling program.
- Establish gold standard datasets, inter-annotator agreement (IAA) targets, and audit sampling processes.
- Serve as the primary user and requirements driver for Label Studio.
- Partner with data engineering to define project configurations, workflows, and pipeline requirements.
- Source, onboard, and manage external labeling vendors and BPOs.
- Set quality SLAs, run calibration sessions, and manage feedback loops to vendors.
- Track operational metrics including label accuracy, cost, and turnaround time to drive improvements.
View Full Description & ApplyYou'll be redirected to the employer's site
