Customer Reliability Engineer

New
A
AndromedaAI infrastructure
Remote / EMEAFull-Time
SalaryCompetitive compensation: + meaningful equity
Apply NowOpens the employer's application page

Job Details

Required Skills
PythonBashKubernetes

Requirements

  • Troubleshoot Linux from the command line using logs, processes, networking, and disk.
  • Have working knowledge of Kubernetes pods, nodes, deployments, services, scheduling, and investigation of failures.
  • Be able to write Python or Bash scripts to automate repetitive work.
  • Explain technical problems clearly to customers in writing.
  • Use sound judgment under pressure, including deciding what to check first and when to escalate.
  • Be willing to participate in a 24/7 on-call rotation.
  • Strong candidates may have production experience with NVIDIA GPUs, drivers, CUDA, DCGM, or the Kubernetes device plugin.
  • Strong candidates may have high-performance networking experience with InfiniBand, RoCE, or NCCL.
  • Strong candidates may have HPC or batch scheduler experience, such as Slurm.
  • Strong candidates may have experience in customer-facing technical roles such as support engineering, TAM, solutions, or professional services.
  • Knowledge of Prometheus, Grafana, Datadog, or similar monitoring tools is advantageous.
  • Experience with Terraform, Ansible, or Helm is advantageous.

Responsibilities

  • Own customer issues from reproduction and diagnosis through resolution or escalation, and close the loop.
  • Debug Linux processes, networking, storage, kernel logs, resource contention, systemd, and journald.
  • Investigate Kubernetes pod, node, scheduling, and resource-limit problems.
  • Troubleshoot GPU failures, including driver and device-plugin issues, XID errors, thermal throttling, and multi-node job failures.
  • Respond to alerts and customer-reported outages, assess impact and severity, and involve the appropriate people.
  • Keep customers updated during incidents and document what happened.
  • Convert incident findings into runbooks, alerts, or fixes.
  • Determine whether faults originate with the provider, Andromeda, or the customer, and pursue provider tickets through resolution.
  • Track recurring provider failures and flag patterns to sourcing decision-makers.
  • Write Python or Bash automation and improve monitoring, health checks, telemetry, dashboards, and alerts.
View Full Description & ApplyYou'll be redirected to the employer's site
Competitive compensation: + meaningful equity
Apply Now