Principal QA Engineer - AI Systems & Platform
H
Hive Financial SystemsAI Operating System
United StatesFull-TimePrincipal
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Experience
- 7+ years of QA engineering experience, with at least 3 years in a lead or senior role
- Required Skills
- PythonExpress.jsSnowflakeTypeScriptAPI testingFastAPINext.jsReactCI/CD
Requirements
- 7+ years of QA engineering experience
- At least 3 years in a lead or senior QA role (writing test code and managing/mentoring engineers)
- Experience leading and growing QA teams, including hiring and setting accountability
- Hands-on experience testing LLM-powered applications, understanding prompt sensitivity and output variance
- Ability to build eval pipelines that catch regressions across model updates
- Automation-first mindset, with Python as a primary tool for writing test code
- Experience building and maintaining CI/CD-integrated test suites
- Comfortable testing complex API chains, async/streaming responses, and multi-service workflows
- Experience with document processing pipelines and knowledge graph outputs
- 0-to-1 mindset: built a QA function from the ground up in an early-stage environment
- Enterprise empathy: tests for confusion and trust failure for non-technical users
- Experience with LLM evaluation frameworks (e.g., LangSmith, PromptFlow, or custom eval pipelines) is a plus
- Experience testing agent frameworks such as LangChain or CrewAI is a plus
- Background in enterprise software or regulated industries where audit trail integrity is non-negotiable is a plus
- Worked alongside Forward Deployed or solutions engineering teams and understand field deployment risk is a plus
Responsibilities
- Establish the testing framework from zero: unit, integration, end-to-end, and LLM-specific evaluation pipelines
- Define quality standards, test coverage requirements, and documentation practices in partnership with the Lead Engineer
- Audit the existing platform and identify highest-risk surfaces before next major customer deployment
- Define the team structure and hiring roadmap, and begin executing against it
- Recruit, hire, and onboard QA engineers as the team grows, setting clear expectations and standards
- Mentor junior and mid-level QA engineers, building their ability to own test domains independently
- Act as the quality culture carrier across the full engineering team
- Report directly to the Lead Engineer and participate in product planning to ensure quality is designed in
- Design evaluation frameworks for non-deterministic LLM outputs, including prompt regression testing and model drift detection
- Build automated test suites for the agent orchestration layer, including governance agent audit trail integrity
- Validate the Enterprise Knowledge Graph for data accuracy, retrieval quality, and failure modes
- Own end-to-end testing of the file ingestion pipeline across document types
- Validate streaming response handling, latency thresholds, and graceful degradation
- Test multi-model routing logic to confirm cost-optimized task allocation behaves correctly
- Partner with the Full-Stack Engineer to define and test trust-layer UX standards
- Build reusable test playbooks for Forward Deployed Engineers to use in new customer deployments
- Act as the internal advocate for the non-technical enterprise user
View Full Description & ApplyYou'll be redirected to the employer's site
