Senior Infrastructure Support Engineer
New
N
NscaleAI Cloud Infrastructure
Location: USFull-TimeSenior
Salary$120,000 — $170,000 USD
Apply NowOpens the employer's application page
Job Details
- Experience
- 6+ years
- Required Skills
- PythonBashLinuxTerraformAnsible
Requirements
- 6+ years in infrastructure, operations, or support engineering in production environments.
- 2–3+ years hands-on experience with GPU, HPC, or large-scale data centre estates.
- Deep technical knowledge of GPU drivers, firmware, and runtime stacks (nvidia-smi, DCGM).
- Hands-on experience with RDMA fabrics, including InfiniBand and RoCE.
- Strong proficiency in Linux system administration and troubleshooting.
- Experience with HPC scheduling using Slurm, including containers via Pyxis/Enroot.
- Solid understanding of L2/L3 networking, routing, BGP, VLANs, and load balancing.
- Proficiency in scripting (Bash, Python) and infrastructure automation tools (Ansible, Terraform).
- Experience building observability dashboards (Prometheus/Grafana) and incident response.
- Strong communication skills for documenting incidents and collaborating with stakeholders.
Responsibilities
- Join the Support duty rotation as a senior escalation point for incidents, investigations, and changes.
- Diagnose and remediate GPU node faults across driver, firmware, and hardware layers.
- Maintain east-west fabric health by running diagnostics on InfiniBand and high-speed Ethernet fabrics.
- Investigate data-path issues on high-performance storage platforms.
- Conduct root cause analysis for major incidents and drive long-term fixes.
- Author and execute changes in live customer environments with risk assessment and peer review.
- Improve dashboards, alerts, and runbooks to reduce repeat incidents.
- Design and implement automation scripts to reduce manual intervention.
- Mentor and upskill mid-level engineers through knowledge sharing and workshops.
- Participate in on-call rotations and travel to sites for onsite technical expertise.
View Full Description & ApplyYou'll be redirected to the employer's site
