- Lead the design and development of our production inference platform.
- Define the technical roadmap for inference infrastructure, model serving, and runtime optimization.
- Build and operate scalable, cost-effective systems for serving large language models in production.
- Evaluate and integrate modern inference technologies, frameworks, and serving runtimes.
- Optimize latency, throughput, GPU utilization, memory efficiency, and infrastructure cost.
- Develop systems for model deployment, traffic routing, autoscaling, scheduling, observability, and operational excellence.
- Partner with ML engineers to productionize new models and inference techniques.
- Establish benchmarking methodologies to evaluate new models, runtimes, and hardware.
- Make key architectural decisions around when to build internally versus leverage open-source or commercial solutions.
- Mentor engineers as the team grows and help establish engineering best practices for AI infrastructure.
PythonC++Go+2 more