Senior Software Engineer, AI Gateway
New
D
DockerAI infrastructure
Work from anywhereFull-TimeSenior
SalaryUnited States: $160,900 – $260,700 + equity.
Apply NowOpens the employer's application page
Job Details
- Experience
- 6+ years of backend engineering experience with production-grade systems
- Required Skills
- KubernetesGoRustDistributed Systems
Requirements
- Have 6+ years of backend engineering experience with production-grade systems.
- Have deep knowledge of distributed and highly scalable systems, cloud-native infrastructure, and API design.
- Have experience building secure, high-throughput services, such as gateways, proxies, load balancers, or policy engines.
- Be fluent in Go and/or Rust; both are preferred.
- Be familiar with AI/ML platforms or model-serving infrastructure.
- Hold a bachelor's degree in Computer Science, Engineering, or a related field, or have equivalent practical experience.
- Prior experience with OpenAI, Anthropic, or similar LLM APIs is a bonus.
- Familiarity with RAG architectures, vector databases, or agent frameworks is a bonus.
- Experience with policy engines, observability frameworks, or API gateways is a bonus.
- Understanding of OAuth2.1, secret management, and cloud IAM systems is a bonus.
- Experience with Kubernetes, Docker, and microservices architecture is a bonus.
Responsibilities
- Design and implement core AI Gateway systems, including the model router, MCP gateway, and control plane.
- Build infrastructure for dynamic model selection and routing, auto-failover, cost-based routing, and policy enforcement.
- Own secure credential storage, session summarization, caching, and rate limiting.
- Develop APIs for developers using OpenAI-compatible interfaces and the Model Context Protocol.
- Build infrastructure for evaluation, telemetry, replay, and backtesting for agents and LLM workflows.
- Tune performance to keep the gateway fast under production load.
- Lead architectural decisions and mentor engineers as the team scales.
- Contribute to roadmap planning, technical strategy, and cross-functional alignment.
View Full Description & ApplyYou'll be redirected to the employer's site
