Junior AI Agent Quality Engineer
New
C
CodeRoadSoftware Development
Latin AmericaFull-TimeJunior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- Advanced English (B2/C1)
- Experience
- 1–3 years
- Required Skills
- PythonJMeterRESTful APIs
Requirements
- Basic understanding of Agentic workflows, prompt engineering (ReAct prompts), and LLM orchestration.
- Familiarity with (or a strong desire to learn) tools like LangSmith, Langfuse, or OpenTelemetry.
- Proficiency in Python 3.10+ for automation and data manipulation.
- Strong experience testing and validating RESTful APIs and JSON structures.
- Advanced English (B2/C1) for global team collaboration and technical documentation.
- Analytical 'breaker' mindset to identify edge cases where AI fails to follow instructions.
- 1–3 years of experience in Software QA (Manual or Automation).
- Exposure to Pytest, Playwright, or Selenium.
- Exposure to LangChain, LangGraph, or Pydantic AI.
- Basic knowledge of Docker and Vector Databases.
- Experience with Locust or JMeter for bulk testing.
Responsibilities
- Create question templates and Python scripts to test how well AI Agents and RAG instances solve complex tasks.
- Evaluate Agent performance using specific metrics: Success Rate, Tool Use Accuracy, Planning Quality, and Autonomy.
- Collaborate with AI Engineers to build Ground Truth datasets and Agentic Task Datasets to benchmark model improvements.
- Write unit and integration tests for Python-based RESTful APIs and agent endpoints.
- Run load and bulk testing (using Locust or JMeter) to see how our AI handles high-volume requests.
- Perform rigorous testing of agent payloads to catch prompt injection risks and logic failures.
View Full Description & ApplyYou'll be redirected to the employer's site
