Member of Technical Staff - Research Infrastructure Engineer
New
B
Black Forest LabsGenerative AI
Depending on your role, you’ll either join us in Freiburg or SF at least 2 days a week (or one full week every other week), or work remotely with a monthly in-person week to stay connected.Full-TimeMiddle
SalaryEU: €100,000 - €230,000 + Equity; US: $150,000 - $300,000 + Equity
Apply NowOpens the employer's application page
Job Details
- Required Skills
- AWSPythonBashGCPKubernetesGoPrometheus
Requirements
- Experience building or operating large-scale training platforms.
- Worked with large scale compute clusters (GPUs).
- Proven ability to debug performance and reliability issues across large distributed fleets.
- Deep knowledge of modern cloud infrastructure including Kubernetes, Infrastructure as Code, AWS, and GCP.
- Experience with SLURM.
- Proficiency in Python, Bash, and Go.
- Experience with Nvidia GPU drivers and operators.
- Experience with OTel and Prometheus.
- Strong problem-solving skills and ability to work independently.
- Strong communication skills and the ability to work effectively with partners.
Responsibilities
- Maintain research infrastructure, ensuring health, and optimizing components to extract peak performance from the system.
- Scale infrastructure to meet growing research demands while maintaining reliability and performance.
- Collaborate with research teams to design solutions that balance performance with cost efficiency.
- Identify and resolve performance bottlenecks and capacity hotspots through deep analysis of distributed systems at scale.
- Build and evolve telemetry and monitoring systems to provide deep visibility into infrastructure performance, utilization, and costs.
- Participate in on-call rotations and incident response to maintain system reliability.
View Full Description & ApplyYou'll be redirected to the employer's site
