Technical Lead Manager, ML Dataset Platform

WaabiAutonomous transportation

Remote US & CanadaFull-TimeManager

Salary not disclosed

Apply NowOpens the employer's application page

Job Details

Have designed and built large-scale data infrastructure for ML training, including storage formats, dataset versioning, ETL, and dataloading.
Understand performance characteristics of petabyte-scale sensor or multimodal data.
Have very strong Python skills and be comfortable with PyTorch training loops.
Have strong software-engineering fundamentals and experience with distributed systems, cloud storage, and data-processing frameworks such as Spark, Ray, or Beam.
Have led a platform or infrastructure effort used by multiple teams, defined its APIs, and driven organizational adoption.
Have managed or tech-led an engineering team, including hiring and mentoring, while balancing hands-on technical work and roadmap ownership.
Communicate clearly with researchers and engineers and build consensus across teams with different needs.
Bonus: Experience with ML data infrastructure in autonomous driving or robotics, including multi-sensor logs, LiDAR, cameras, or HD maps.
Bonus: Experience with columnar or tensor storage formats such as Parquet, Lance, WebDataset, or Zarr, and dataset versioning tools.
Bonus: Experience with multi-cloud or hybrid training environments, data lineage, metadata catalogs, or feature-store systems.

Lead and grow the ML Dataset Platform team, setting technical direction and roadmap, hiring, and mentoring engineers.
Define APIs, tools, and storage formats for ML datasets across training workloads.
Build dataset-management tooling for versioning and efficient incremental updates.
Design APIs for training-time transformations, from ahead-of-time ETL to dynamic data-loader transformations.
Own common dataloading tools and provide observability into bottlenecks and training-job throughput.
Streamline data access for multi-cloud training with predictable cost and latency.
Make data lineage automatic across datasets, from raw data through trained models.
Partner with the ML Training Platform team and ML development teams on platform boundaries and shared tooling.
Work with model teams to migrate training data to the platform and feed findings into the roadmap.

View Full Description & ApplyYou'll be redirected to the employer's site