Forward Deployed Infrastructure Engineer

Hyperbolic LabsAI infrastructure

Forward Deployed Infrastructure Engineer - Eastern US, EASTERN TIME ZONEFull-Time

Salary not disclosed

Apply NowOpens the employer's application page

Job Details

Have deep Linux experience and be comfortable working in the CLI, including in broken environments.
Bring hands-on multinode GPU experience, including NCCL, InfiniBand or RoCE, collective performance debugging, topology, and placement.
Be able to triage GPU-node hardware faults, including XID and ECC errors, and use dmesg, lspci, and nvidia-smi.
Have experience provisioning and operating GPU clusters with Kubernetes, Slurm, or both.
Be comfortable working directly with customers, including delivering bad news and attributing faults confidently.
Use sound judgment to decide when to continue investigating and when to escalate.
Preferred: Experience with parallel filesystems such as Weka, Lustre, or GPFS and high-performance storage.
Preferred: Production experience with Grafana, Prometheus, or similar observability stacks.
Preferred: Prior forward deployed engineer, solutions architect, or technical account manager experience.
Preferred: Infrastructure as code experience with Terraform or Ansible, and CI for cluster provisioning.
Preferred: Practitioner-side exposure to model training or inference workloads.

Build, validate, and benchmark new customer clusters, then hand them over with documentation.
Debug multinode and NCCL performance issues, GPU and hardware faults, driver and fabric issues, and container and scheduler problems.
Coordinate provider escalations involving maintenance windows, RMAs, hung nodes, and disputed fault attribution.
Own named customer accounts, learn their workloads, and advise them on infrastructure changes.
Own monitoring and alerting for customer clusters using Grafana, Prometheus, or similar tools.
Automate repeat work and turn escalations into runbooks for the L1 tier.

View Full Description & ApplyYou'll be redirected to the employer's site