Senior QA Analyst, AI & Threat Analytics

3+ years of experience in QA, test automation or evaluation of complex systems Hands-on experience writing Python automation or evaluation scripts for testing APIs, model outputs or data pipelines Understanding of machine learning concepts (classification, summarization, embeddings, anomaly detection, behavior analysis) Experience testing LLM-integrated features or AI-driven workflows Familiarity with ML tooling such as Pytest, Hugging Face libraries, Jupyter Notebooks or similar Experience with API testing, JSON validation and test data generation Ability to test nondeterministic systems and evaluate model quality with structured methodologies Strong analytical, troubleshooting and communication skills Experience testing cloud-integrated applications or systems deployed on AWS Bachelor’s degree in Computer Science, Information Systems or related field, or equivalent practical experience

Design and execute test plans for AI and ML-driven features Develop Python-based automation and evaluation scripts Test LLM-based features for accuracy, guardrail effectiveness, and behavior under edge cases Validate data pipelines, input/output transformations and model integration points Conduct regression testing on ML models and AI workflows Test enterprise-scale behavior analysis features Validate security, privacy and zero-trust requirements for AI model interactions Collaborate with ML engineers and cross-functional teams to refine model behavior Analyze logs, metrics and model outputs to detect anomalies