- Own end-to-end inference performance across the platform, with clear responsibility for latency, throughput, and reliability targets
- Lead the architecture and design of core inference systems, including request routing, async execution, queuing, GPU scheduling, and result delivery
- Drive the platform toward sub-1 second inference where feasible, identifying bottlenecks across networking, services, storage, and GPU execution
- Make high-impact architectural decisions with performance, scalability, and operational simplicity as first-class concerns
- Partner with ML and model teams to ensure models are production-ready from a performance perspective
- Define performance budgets, SLAs, and success metrics
- Lead deep-dive investigations into latency spikes and system-level performance issues
- Influence and mentor engineers across teams on performance engineering and distributed systems
- Improve tooling, observability, and profiling capabilities
- Advocate for pragmatic engineering best practices around testing, benchmarking, and documentation
PHPPythonGo+2 more