Senior Manager, Machine Learning (Data Operations)
C
CoalitionCyber Insurance
Any location, CanadaFull-TimeManager
Salary123,100 - 186,800 CAD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonSQLMachine Learning
Requirements
- 5+ years in ML data operations, data labeling, or a related field.
- Deep understanding of annotation quality frameworks: IAA, consensus labeling, gold standard evaluation, error taxonomy, and calibration workflows.
- Direct experience managing labeling platforms (Label Studio strongly preferred; Scale AI, Labelbox, Prodigy, or similar acceptable).
- Track record managing outsourced labeling vendors or BPOs for ML data production.
- Familiarity with common ML labeling tasks: text classification, NER, document extraction, intent detection.
- Proficiency in Python and SQL.
- Strong opinions on what makes labeled data good or bad and the willingness to push back on low-quality standards.
Responsibilities
- Define annotation guidelines, taxonomies, and edge-case protocols for each labeling program.
- Establish gold standard datasets, inter-annotator agreement (IAA) targets, and audit sampling processes.
- Identify and remediate mislabeled data in existing datasets.
- Manage the Label Studio platform and define project configuration needs and workflow designs.
- Partner with cross-functional teams to translate model requirements into well-structured labeling tasks.
- Source, onboard, and manage external labeling vendors and BPOs to ensure quality and throughput.
- Track operational metrics such as label accuracy, IAA scores, cost per label, and turnaround time.
View Full Description & ApplyYou'll be redirected to the employer's site
