Senior Software Engineer (AI Inference & Runtime Platform)
New
J
JobgetherAI infrastructure
Based in the United StatesFull-TimeSenior
Salary$140,000–$230,000
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 5 years of experience delivering production systems using a systems programming language.
- Required Skills
- PythonKubernetesFastAPIRustTerraformDistributed Systems
Requirements
- Have at least 5 years of experience delivering production systems using a systems programming language.
- Bring strong Rust expertise, or deep Go, C/C++, or Zig experience and a genuine willingness to work in Rust.
- Understand asynchronous runtimes, memory safety, and low-level debugging, including troubleshooting at the system-call boundary.
- Have hands-on experience operating Kubernetes workloads, including controllers or operators, scheduling, autoscaling, and node lifecycle management.
- Be able to threat-model isolation boundaries involving namespaces, cgroups, seccomp, and hypervisors.
- Know secure AI-agent execution practices, including least-privilege access, sandbox credential protection, audit trails, and human approval for changes to systems of record.
- Have experience deploying or operating open-weight LLM inference infrastructure such as vLLM or SGLang, and packaging models as metered production endpoints.
- Demonstrate distributed-systems performance skills, including measurement-driven optimization and latency analysis.
- Have practical experience with relevant tools and technologies, including Rust, Tokio, Python, FastAPI, Kubernetes operators, controller-runtime, Kubebuilder, CRDs, KEDA, Karpenter, GPU device plugins, or DRA.
- Be familiar with Firecracker, Kata Containers, gVisor, or equivalent isolation technologies.
- Have experience with secrets management, egress controls, stateful technologies such as pgvector, Qdrant, or Neo4j, and backup and failover practices.
- Be comfortable operating across AWS, Azure, GCP, and managed GPU cloud environments, using Terraform/OpenTofu, Bicep, and observability tooling such as OpenTelemetry.
- Hold a bachelor's degree; a master's degree is a plus.
- Be able to travel twice a year for company summits.
- Complete a written take-home assignment followed by a live, two-hour technical session with the engineering team.
Responsibilities
- Manage open-weight large language model serving infrastructure, including deployment, configuration, cold-start optimization, SLOs, upgrades, and canary releases.
- Operate and improve Kubernetes infrastructure for inference and sandbox workloads, including operators, CRDs, autoscaling, GPU scheduling and sharing, and node lifecycle management.
- Own the stateful data plane, document recovery procedures, and regularly test restoration processes.
- Develop and maintain the secure agent-sandboxing runtime and host-side control plane, including lifecycle management, command execution, snapshots, forking, teardown, usage metering, and isolation-boundary security.
- Build and maintain FastAPI control-plane services, infrastructure-as-code configurations using Terraform/OpenTofu and Bicep, and operational dashboards.
- Provide infrastructure for backend engineering teams and investigate issues spanning platform and application layers.
- Establish and maintain observability, performance, reliability, and operational practices.
- Contribute reviewed pull requests, comprehensive documentation, reproducible builds, and releases to an open-source-oriented development process.
View Full Description & ApplyYou'll be redirected to the employer's site
