Senior AI Engineer - Observability
New
J
JobgetherAI / Software Engineering
Fully remote position within the United States.Full-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonCloud ComputingC#Data engineering.NETCI/CDPrompt EngineeringLLM
Requirements
- 5+ years of professional software engineering experience building and supporting production systems.
- Hands-on experience developing or operating LLM-powered features, RAG systems, or AI workflows.
- Strong programming capabilities in Python, C#/.NET, or both.
- Practical understanding of prompts, embeddings, vector search, and modern LLM application architecture.
- Demonstrated experience designing evaluations, quality metrics, or regression frameworks.
- Strong observability fundamentals covering tracing, logging, metrics, alerting, and production debugging.
- Experience with CI/CD, Git-based workflows, and cloud environments.
- Strong communication, collaboration, and stakeholder management skills.
- A strong product mindset, curiosity about AI behavior, and commitment to quality.
Responsibilities
- Partner with product engineering teams to design, build, and enhance AI-powered features using LLMs, RAG, semantic search, and agentic workflows.
- Contribute directly to production codebases using Python, C#/.NET, or related technologies.
- Design and implement AI evaluation pipelines, scoring methodologies, regression frameworks, and quality gates.
- Build and maintain versioned golden datasets covering real-world use cases, edge cases, and failure modes.
- Instrument LLM interactions, RAG pipelines, tool calls, and agent workflows using observability platforms.
- Track and analyze latency, token usage, cost, retrieval quality, groundedness, and safety signals.
- Develop reusable libraries, SDKs, templates, and documentation for AI instrumentation and evaluation.
- Coach product teams on AI quality, observability, and reliable release practices.
- Support responsible AI delivery aligned with security, privacy, SOC 2, and ISO 27001 standards.
View Full Description & ApplyYou'll be redirected to the employer's site
