Staff Software Engineer, AI Inference
New
S
SylloLegal Technology
United States - RemoteFull-TimeStaff
Salary$190- $230K
Apply NowOpens the employer's application page
Job Details
- Required Skills
- PythonC++GoRustLLM
Requirements
- Significant experience designing and operating production AI inference systems.
- Experience building or leading production LLM serving infrastructure.
- Deep experience with one or more modern inference runtimes and frameworks such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, Hugging Face TGI, NVIDIA Dynamo, or comparable technologies.
- Strong background in distributed systems, backend infrastructure, or high-performance platform engineering.
- Experience optimizing inference performance across GPU workloads, including latency, throughput, batching, memory utilization, and serving efficiency.
- Experience operating GPU infrastructure in production.
- Strong proficiency in Python and at least one systems programming language (such as Go, Rust, or C++).
- Proven ability to lead technical architecture for complex infrastructure initiatives.
- Excellent communication skills and the ability to influence technical direction across engineering teams.
Responsibilities
- Lead the design and development of our production inference platform.
- Define the technical roadmap for inference infrastructure, model serving, and runtime optimization.
- Build and operate scalable, cost-effective systems for serving large language models in production.
- Evaluate and integrate modern inference technologies, frameworks, and serving runtimes.
- Optimize latency, throughput, GPU utilization, memory efficiency, and infrastructure cost.
- Develop systems for model deployment, traffic routing, autoscaling, scheduling, observability, and operational excellence.
- Partner with ML engineers to productionize new models and inference techniques.
- Establish benchmarking methodologies to evaluate new models, runtimes, and hardware.
- Make key architectural decisions around when to build internally versus leverage open-source or commercial solutions.
- Mentor engineers as the team grows and help establish engineering best practices for AI infrastructure.
View Full Description & ApplyYou'll be redirected to the employer's site
