Customer Reliability Engineer
New
A
AndromedaAI infrastructure
Remote / EMEAFull-Time
SalaryCompetitive compensation: + meaningful equity
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonBashKubernetes
Requirements
- Troubleshoot Linux from the command line using logs, processes, networking, and disk.
- Have working knowledge of Kubernetes pods, nodes, deployments, services, scheduling, and investigation of failures.
- Be able to write Python or Bash scripts to automate repetitive work.
- Explain technical problems clearly to customers in writing.
- Use sound judgment under pressure, including deciding what to check first and when to escalate.
- Be willing to participate in a 24/7 on-call rotation.
- Strong candidates may have production experience with NVIDIA GPUs, drivers, CUDA, DCGM, or the Kubernetes device plugin.
- Strong candidates may have high-performance networking experience with InfiniBand, RoCE, or NCCL.
- Strong candidates may have HPC or batch scheduler experience, such as Slurm.
- Strong candidates may have experience in customer-facing technical roles such as support engineering, TAM, solutions, or professional services.
- Knowledge of Prometheus, Grafana, Datadog, or similar monitoring tools is advantageous.
- Experience with Terraform, Ansible, or Helm is advantageous.
Responsibilities
- Own customer issues from reproduction and diagnosis through resolution or escalation, and close the loop.
- Debug Linux processes, networking, storage, kernel logs, resource contention, systemd, and journald.
- Investigate Kubernetes pod, node, scheduling, and resource-limit problems.
- Troubleshoot GPU failures, including driver and device-plugin issues, XID errors, thermal throttling, and multi-node job failures.
- Respond to alerts and customer-reported outages, assess impact and severity, and involve the appropriate people.
- Keep customers updated during incidents and document what happened.
- Convert incident findings into runbooks, alerts, or fixes.
- Determine whether faults originate with the provider, Andromeda, or the customer, and pursue provider tickets through resolution.
- Track recurring provider failures and flag patterns to sourcing decision-makers.
- Write Python or Bash automation and improve monitoring, health checks, telemetry, dashboards, and alerts.
View Full Description & ApplyYou'll be redirected to the employer's site