Senior Software Engineer (Python) - Agent Evaluation
New
M
MindriftAI Systems Evaluation
Open to candidates in the North America, South America, Asia and Europe.ContractSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Languages
- English (B2 level or higher)
- Experience
- 5+ years
- Required Skills
- DockerPostgreSQLPythonTypeScriptFastAPIReactCI/CD
Requirements
- 5+ years of professional experience with Python.
- Strong experience with FastAPI, pytest, and async/await.
- Hands-on experience with Docker, PostgreSQL, and CI/CD pipelines.
- Proven experience writing and maintaining automated tests.
- English proficiency at B2 level or higher.
- Availability to work 30+ hours per week.
- Ability to complete a technical assessment.
- Willingness to complete identity verification and a background check.
- Willingness to use Discord for project communication.
- Full-stack experience with React and TypeScript is a plus.
Responsibilities
- Build realistic developer environments including codebase, infrastructure, tickets, docs, and conversations.
- Design complex tasks from intermediate environment states, including prompt creation and task solvability definition.
- Write automated tests to verify agent-generated solutions, ensuring criteria are neither too strict nor too lenient.
- Iterate on tasks and tests based on QA feedback and analysis of agent failure cases.
- Analyze and evaluate how AI agents handle real-world developer tasks.
View Full Description & ApplyYou'll be redirected to the employer's site
