Senior Staff Engineer - Infrastructure Systems

New
R
RunwareAI infrastructure
Workable workplace: remote; Workable locations: California, United StatesFull-TimeStaff
Salary not disclosed
Apply NowOpens the employer's application page

Job Details

Experience
12+ years building and operating systems software, with several of those on GPU software running in production at scale.
Required Skills
PythonKubernetesPyTorch

Requirements

  • Have 12+ years building and operating systems software, including several years of production GPU software experience.
  • Bring hands-on knowledge of the NVIDIA kernel driver and modules, CUDA runtime and libraries, NVML, DCGM, and Container Toolkit, including version interactions.
  • Have a track record of tracing GPU failures across hardware, firmware, drivers, runtimes, and applications, including Xid, ECC, PCIe, NVLink, thermal, and memory faults.
  • Know Linux internals, including the kernel, device drivers, PCIe, NUMA, IOMMU, cgroups, and containers.
  • Use diagnostic tools such as perf, eBPF, and ftrace.
  • Write code daily; be fluent in Python and at least one of C, C++, Rust, or Go.
  • Be comfortable using CUDA to write tests or reproduce bugs.
  • Have experience running GPUs on Kubernetes and bare metal, including device plugins, the GPU Operator, MIG, image builds, and provisioning at scale.
  • Understand GPU use in inference frameworks such as PyTorch, TensorRT, and ONNX Runtime, including memory allocation, compilation, caches, and multi-GPU execution.
  • Write technical design documents, postmortems, and ticket updates that others can act on.
  • Use AI to accelerate work and have a clear view of how a small team should operate with these tools.
  • Be comfortable owning outcomes and prioritizing work in a fast-moving, ambiguous environment.
  • Nice to have: experience with GPU drivers, CUDA libraries, or GPU system software; Blackwell hardware, NVLink, NVSwitch, or liquid-cooled racks; and multi-GPU or multi-node execution using NCCL, GPUDirect, or RDMA.
  • Nice to have: firmware qualification, inference serving internals, non-NVIDIA accelerators, production open-source projects, multi-tenant GPU isolation, confidential computing, or 24x7 services with published service levels.

Responsibilities

  • Own versioned, reproducible combinations of the kernel, NVIDIA driver, firmware, CUDA runtime and libraries, and container toolkit.
  • Qualify driver and CUDA releases across GPU generations, frameworks, and model families, catching correctness, performance, and memory regressions.
  • Diagnose GPU failures and determine whether their cause is hardware, firmware, driver, runtime, or application software.
  • Build GPU acceptance and diagnostics tests, including burn-in, DCGM diagnostics, memory and interconnect tests, and real-workload tests.
  • Integrate DCGM, driver, kernel-log, and BMC telemetry; classify GPU faults and automate drain, reset, and replacement paths.
  • Own GPU workload exposure through container runtimes, device plugins, the GPU Operator, MIG, sharing policies, and tenant isolation.
  • Bring up new GPU generations and pods with hardware and node operations teams, validating drivers, firmware, NVLink, and NVSwitch.
  • Benchmark image, video, audio, and LLM inference across GPU generations and software stacks, and feed results into placement, pricing, and purchasing.
  • Work with NVIDIA and upstream projects to report and resolve driver and firmware bugs and contribute fixes to open-source tools.
  • Use AI agents for log and Xid triage, test generation, release qualification, and runbook automation.
View Full Description & ApplyYou'll be redirected to the employer's site
View details
Apply Now