Senior Staff Engineer - Infrastructure Systems
New
R
RunwareAI infrastructure
Workable workplace: remote; Workable locations: California, United StatesFull-TimeStaff
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 12+ years building and operating systems software, with several of those on GPU software running in production at scale.
- Required Skills
- PythonKubernetesPyTorch
Requirements
- Have 12+ years building and operating systems software, including several years of production GPU software experience.
- Bring hands-on knowledge of the NVIDIA kernel driver and modules, CUDA runtime and libraries, NVML, DCGM, and Container Toolkit, including version interactions.
- Have a track record of tracing GPU failures across hardware, firmware, drivers, runtimes, and applications, including Xid, ECC, PCIe, NVLink, thermal, and memory faults.
- Know Linux internals, including the kernel, device drivers, PCIe, NUMA, IOMMU, cgroups, and containers.
- Use diagnostic tools such as perf, eBPF, and ftrace.
- Write code daily; be fluent in Python and at least one of C, C++, Rust, or Go.
- Be comfortable using CUDA to write tests or reproduce bugs.
- Have experience running GPUs on Kubernetes and bare metal, including device plugins, the GPU Operator, MIG, image builds, and provisioning at scale.
- Understand GPU use in inference frameworks such as PyTorch, TensorRT, and ONNX Runtime, including memory allocation, compilation, caches, and multi-GPU execution.
- Write technical design documents, postmortems, and ticket updates that others can act on.
- Use AI to accelerate work and have a clear view of how a small team should operate with these tools.
- Be comfortable owning outcomes and prioritizing work in a fast-moving, ambiguous environment.
- Nice to have: experience with GPU drivers, CUDA libraries, or GPU system software; Blackwell hardware, NVLink, NVSwitch, or liquid-cooled racks; and multi-GPU or multi-node execution using NCCL, GPUDirect, or RDMA.
- Nice to have: firmware qualification, inference serving internals, non-NVIDIA accelerators, production open-source projects, multi-tenant GPU isolation, confidential computing, or 24x7 services with published service levels.
Responsibilities
- Own versioned, reproducible combinations of the kernel, NVIDIA driver, firmware, CUDA runtime and libraries, and container toolkit.
- Qualify driver and CUDA releases across GPU generations, frameworks, and model families, catching correctness, performance, and memory regressions.
- Diagnose GPU failures and determine whether their cause is hardware, firmware, driver, runtime, or application software.
- Build GPU acceptance and diagnostics tests, including burn-in, DCGM diagnostics, memory and interconnect tests, and real-workload tests.
- Integrate DCGM, driver, kernel-log, and BMC telemetry; classify GPU faults and automate drain, reset, and replacement paths.
- Own GPU workload exposure through container runtimes, device plugins, the GPU Operator, MIG, sharing policies, and tenant isolation.
- Bring up new GPU generations and pods with hardware and node operations teams, validating drivers, firmware, NVLink, and NVSwitch.
- Benchmark image, video, audio, and LLM inference across GPU generations and software stacks, and feed results into placement, pricing, and purchasing.
- Work with NVIDIA and upstream projects to report and resolve driver and firmware bugs and contribute fixes to open-source tools.
- Use AI agents for log and Xid triage, test generation, release qualification, and runbook automation.
View Full Description & ApplyYou'll be redirected to the employer's site