Staff/Principal Software Engineer, AI Platform Model Serving
New
J
JobgetherAI infrastructure
Based in the United States, with remote work flexibilityFull-TimePrincipal
SalaryAnnual salary range of $890,000–$1,690,000
Apply NowOpens the employer's application page
Job Details
- Experience
- At least 12 years of relevant experience in AI/ML infrastructure, distributed systems, or related engineering domains
- Required Skills
- AWSKubernetesA/B testingDistributed Systems
Requirements
- Hold a bachelor’s, master’s, or doctoral degree in Computer Science, Applied Mathematics, Engineering, or a related field.
- Have at least 12 years of relevant experience in AI/ML infrastructure, distributed systems, or related engineering domains.
- Bring extensive experience designing and operating high-traffic distributed services, high-performance RPC systems, and production infrastructure.
- Have demonstrated experience building and scaling inference platforms for machine learning and foundation models.
- Understand latency and throughput optimization, resource allocation, GPU/CPU utilization, and inference efficiency.
- Have practical experience with model-serving platforms or runtimes such as NVIDIA Triton Inference Server, vLLM, SGLang, TensorRT-LLM, TGI, DeepSpeed-Inference, or NVIDIA Dynamo.
- Have experience deploying and scaling AI workloads with Kubernetes and a public cloud platform such as AWS, Microsoft Azure, or Google Cloud Platform.
- Be familiar with model gateways, intelligent routing, control-plane architecture, experimentation infrastructure, and A/B testing for production AI systems.
- Be able to set multi-year technical roadmaps, lead complex architectural initiatives, and influence decisions across teams.
- Communicate technical concepts to non-technical stakeholders and produce design proposals, architecture decision records, and technical documentation.
- Work with product managers, operations teams, and senior leadership to translate business objectives into scalable, maintainable technical solutions.
Responsibilities
- Define and drive the architecture of a unified AI model-serving platform integrating internal infrastructure, third-party hosting, and external model providers.
- Design and optimize distributed systems for large-scale production workloads, including predictive models and generative AI applications.
- Develop techniques such as disaggregated serving, speculative decoding, prompt caching, intelligent routing, and multi-GPU inference to reduce latency and cost.
- Enable teams to deploy, route, evaluate, and A/B test models across runtimes, infrastructure environments, and providers.
- Design reliable, resumable execution systems for long-running AI agents and asynchronous inference tasks.
- Evaluate emerging AI-serving technologies, identify infrastructure bottlenecks, and turn research and experimentation into production solutions.
- Own architectural decisions and multi-year roadmaps for the platform.
- Build alignment across teams through documentation, architecture reviews, and evidence-based decisions; mentor engineers and raise engineering standards.
- Collaborate with product, infrastructure, operations, and executive teams on technical trade-offs, business priorities, reliability, and infrastructure costs.
View Full Description & ApplyYou'll be redirected to the employer's site
