Member of Technical Staff - Large Scale Data Infrastructure
New
B
Black Forest LabsGenerative AI
Depending on your role, you’ll either join us in Freiburg or SF at least 2 days a week (or one full week every other week), or work remotely with a monthly in-person week to stay connected.Full-TimeMiddle
SalaryEU: €100,000 - €230,000 + Equity; US: $150,000 - $300,000 + Equity
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonKubernetesPyTorchDistributed Systems
Requirements
- Proven experience building and operating data pipelines at petabyte scale.
- Expertise in optimizing data loading processes.
- Experience working with petabyte-scale video and image datasets.
- Proficiency writing processing jobs operating on millions of files.
- Ability to debug distributed system bottlenecks across large fleets of machines.
- Proficiency in Python and PyTorch DataLoader internals.
- Experience with object storage systems such as S3, Azure Blob, or GCS.
- Familiarity with Parquet for metadata.
- Knowledge of video tools like ffmpeg and PyAV.
- Nice to have: Experience with streaming dataset formats like WebDataset.
- Nice to have: Understanding of video codec internals and frame-accurate seeking.
- Nice to have: Experience with Slurm and Kubernetes for job orchestration.
Responsibilities
- Develop scalable data loaders for training runs across thousands of GPUs.
- Build efficient storage and retrieval systems for petabyte-scale datasets.
- Implement multi-cloud object storage abstraction.
- Execute large-scale data migrations across storage systems and providers.
- Debug and resolve performance bottlenecks in distributed data loading.
View Full Description & ApplyYou'll be redirected to the employer's site
