Senior HPC Engineer - Fleet Engineering
New
L
LambdaAI cloud infrastructure
Remote, USAFull-TimeSenior
Salary227,000 - 356,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Experience
- 7+ years of experience in Site Reliability Engineering, HPC Engineering, DevOps, or a similar role
- Required Skills
- PythonGoPrometheusLinuxTerraformAnsible
Requirements
- Have 7+ years of experience in Site Reliability Engineering, HPC Engineering, DevOps, or a similar role.
- Have a strong understanding of modern AI infrastructure, GPU architectures, and hardware performance optimization.
- Have a strong understanding of Linux-based systems in a distributed environment.
- Have experience configuring and troubleshooting InfiniBand (IB), RoCE, CLOS fabrics, 100GbE, Ethernet/switching, GPU-direct, and NCCL environments.
- Have a solid understanding of Python and Go, and experience working with software engineering teams to improve internal tooling.
- Have experience with monitoring and alerting tools such as Prometheus, Grafana, or ClickHouse.
- Be proficient in automation and configuration management tools such as Ansible and Terraform.
- Have experience with machine learning or deep learning frameworks such as PyTorch or TensorFlow, and benchmarking tools such as DeepSpeed or MLPerf.
- Have knowledge of containerization and orchestration technologies such as Docker and Kubernetes.
- Have experience building or operating HPC resources.
Responsibilities
- Build and operate monitoring and alerting for cluster health across fabric, GPU, power/thermal, and job-level signals.
- Remotely deploy and configure large-scale HPC clusters for AI workloads, using automation wherever possible.
- Automate cluster lifecycle management for operating systems, firmware, drivers, and networking using Ansible and Terraform.
- Create runbooks and automated remediations for common cluster failure modes for Support and HPC Support.
- Troubleshoot cluster issues across InfiniBand/RoCE, NCCL, GPU-direct, fabric, switching, and power with on-site deployment teams.
- Participate in on-call rotations and lead incident response for cluster-level problems.
- Maintain Standard Operating Procedures and provide engineering teams with requirements for simplification, stability, and operational efficiency.
View Full Description & ApplyYou'll be redirected to the employer's site
