Senior Manager, Machine Learning (Data Operations)
New
C
CoalitionCyber insurance
Listing locations: CanadaFull-TimeManager
Salary123,100 - 153,900 CAD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years in ML data operations, data labeling, or a related field (ML engineering, data science, or data engineering with heavy labeling exposure)
- Required Skills
- PythonSQL
Requirements
- Have 5+ years in ML data operations, data labeling, or a related field such as ML engineering, data science, or data engineering with substantial labeling exposure.
- Understand annotation quality frameworks, including IAA, consensus labeling, gold-standard evaluation, error taxonomy, and calibration workflows.
- Have direct experience managing labeling platforms; Label Studio is strongly preferred, with Scale AI, Labelbox, Prodigy, or similar platforms also acceptable.
- Have a track record managing outsourced labeling vendors or BPOs for ML data production.
- Be familiar with ML labeling tasks such as text classification, NER, document extraction, and intent detection.
- Be comfortable working in Python and SQL.
- Insurance, cybersecurity, or fintech experience is a plus but not required.
Responsibilities
- Define annotation guidelines, taxonomies, edge-case protocols, gold-standard datasets, inter-annotator agreement targets, and audit sampling processes.
- Identify and remediate mislabeled data in existing datasets.
- Drive Label Studio project configurations, workflow designs, pre-labeling pipeline requirements, and integrations with ML infrastructure.
- Partner with data engineering on the labeling platform.
- Work with ML engineers, data scientists, and product managers to translate model requirements into structured labeling tasks.
- Challenge task designs when instructions are ambiguous or likely to produce unreliable labels.
- Source, onboard, and manage external labeling vendors and BPOs with the operations team.
- Set quality SLAs, run calibration sessions, and manage feedback loops with vendors and labelers.
- Define and track label accuracy, IAA scores, cost per label, and turnaround time.
- Identify opportunities for active learning, model-assisted labeling, and pre-annotation to reduce cost without sacrificing quality.
View Full Description & ApplyYou'll be redirected to the employer's site
