GenAI Platform Engineer – LLM Gateway

New
J
JobgetherGenAI infrastructure
Fully remote working arrangement from IndiaFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page

Job Details

Languages
Excellent written and verbal communication skills in English
Experience
At least 9 years of professional software engineering experience, including a minimum of 3 years developing and delivering production-grade GenAI or LLM-based applications and platforms.
Required Skills
AWSDockerPythonFastAPIRESTful APIsGitHub Actions

Requirements

  • Bachelor's or Master's degree in Computer Science, Computer Applications, Engineering, or a related discipline, or equivalent practical experience.
  • At least 9 years of professional software engineering experience, including a minimum of 3 years developing and delivering production-grade GenAI or LLM-based applications and platforms.
  • Strong proficiency in Python backend development, including asynchronous programming, FastAPI or similar frameworks, REST APIs, and streaming responses.
  • Hands-on experience integrating LLM provider APIs such as OpenAI, Azure OpenAI, and AWS Bedrock.
  • Experience implementing tool calling, function calling, and integrations across multiple LLM providers.
  • Experience deploying and operating services on AWS using containerization technologies and CI/CD pipelines.
  • Practical knowledge of Docker, GitHub Actions, and automated testing in software delivery workflows.
  • Strong understanding of backend architecture, scalable services, API integrations, and production software engineering practices.
  • Ability to troubleshoot complex production issues, conduct root-cause analysis, and implement long-term solutions.
  • Excellent written and verbal communication skills in English and ability to collaborate across globally distributed teams.
  • Experience with AI/LLM gateways or API gateways, model routing, API key management, access controls, or cost tracking is preferred.
  • Knowledge of agentic AI architectures and Model Context Protocol (MCP), and experience with LangGraph, LangChain, or similar frameworks, are preferred.

Responsibilities

  • Design, develop, and operate enterprise LLM Gateway features primarily using Python.
  • Build and maintain multi-provider model integrations, intelligent routing, failover, API key management, and access controls.
  • Develop usage metering, budget management, cost transparency, and chargeback reporting features.
  • Integrate agentic AI capabilities and develop MCP servers, cross-provider tool calling, and agent-to-agent connectivity.
  • Maintain open-source platform foundations, assess upstream releases, manage custom extensions, and deliver upgrades without regressions.
  • Build automated quality gates in CI/CD, including unit, integration, contract, and performance testing.
  • Implement observability using metrics, distributed tracing, and logging tools such as OpenTelemetry and Langfuse.
  • Monitor platform performance and resolve latency, reliability, error-rate, and infrastructure-cost issues.
  • Participate in production support and on-call rotations, investigate root causes, resolve incidents, and conduct blameless postmortems.
  • Support internal engineering teams with technical guides, runbooks, code samples, reusable AI agent skills, and onboarding.
View Full Description & ApplyYou'll be redirected to the employer's site
View details
Apply Now