Senior QA Engineer - AI
New
D
DeltaFinancial Technology
IndiaFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 4-6 years of QA / SDET experience; at least 1 year owning quality for AI products.
- Required Skills
- PythonAPI testing
Requirements
- At least 1 year owning quality for AI products (chatbots, agents, content generation) in a lead or primary-owner capacity.
- Hands-on experience building or operating AI evaluation harnesses (e.g., DeepEval, RAGAS, or Promptfoo).
- 4-6 years of professional experience in QA or SDET roles.
- Working proficiency in Python for tooling and trace analysis.
- Strong API testing experience.
- Ability to use observability tools like LangSmith, Langfuse, or Opik to perform root cause analysis.
- Strong engineering skills to build internal tooling and automation.
- Clear written communication for documenting engineering-actionable findings.
- Familiarity with trading, crypto derivatives, or margin flows is a plus.
Responsibilities
- Build and maintain golden datasets for AI products, verified against a documented source of truth.
- Design layered scoring for AI outputs using both deterministic rules and model-graded evaluation.
- Manage evaluation runs end-to-end to compare release candidates against prior baselines.
- Own and operate the release gate for approving prompt or model changes for production.
- Identify and mitigate failure modes including hallucinations, incorrect tool selection, and PII exposure.
- Investigate production traces to determine root causes of failures across retrieval, generation, and tool usage.
- Convert findings into actionable engineering evidence and regression test cases.
- Define and report quality metrics for AI surfaces to drive product improvements.
View Full Description & ApplyYou'll be redirected to the employer's site
