Senior AI Engineer - Observability
New
O
OnBoardBoard Management Software
Remote - United StatesFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 5+ years
- Required Skills
- PythonC#.NETCI/CDAzure DevOps
Requirements
- 5+ years of software engineering experience building production systems.
- Hands-on experience building or operating LLM-powered features, RAG systems, or AI workflows.
- Strong engineering ability in Python, C#/.NET, or both.
- Practical understanding of prompts, embeddings, vector search, and retrieval quality.
- Experience designing evaluations, quality metrics, or regression frameworks for AI or software systems.
- Strong observability fundamentals including tracing, logging, metrics, and alerting.
- Experience with CI/CD, git workflows, and cloud production release practices (Azure DevOps preferred).
- Ability to communicate effectively with engineering, product, QA, and security stakeholders.
- Proficiency with AI-assisted development tools such as Claude Code or PlayerZero.
- Strong product judgment and curiosity about AI system behavior with real users.
Responsibilities
- Partner with product engineers to design and improve AI features using LLMs, RAG, semantic search, and agentic workflows.
- Contribute to production codebases in Python, C#/.NET, or related technologies.
- Design and implement evaluation pipelines, scoring methodologies, and versioned golden datasets for AI features.
- Implement observability for LLM interactions and RAG pipelines using platforms like Arize, Langfuse, or OpenTelemetry.
- Define quality gates, thresholds, and regression testing to determine when AI features are ready to ship.
- Analyze production traces to identify quality issues, cost spikes, and performance regressions.
- Build reusable libraries, SDKs, and templates to scale AI instrumentation and evaluation across product teams.
- Ensure AI systems align with security, privacy, SOC 2, and data-residency requirements.
View Full Description & ApplyYou'll be redirected to the employer's site
