Senior QA Analyst, AI & Machine Learning Systems
Company:Keeper Security
Location:United States, Canada, El Dorado Hills CA, Chicago IL, EST, PST
Languages:English
Seniority level:Senior, 3+ years
Experience:3+ years
Skills:
AWSPythonArtificial IntelligenceCloud ComputingCybersecurityData AnalysisMachine LearningNumpyQA AutomationAPI testingRegression testingPandasJSON
Requirements:
3+ years of experience in QA, test automation, or evaluation of complex software systems Hands-on experience writing Python automation or evaluation scripts Understanding of machine learning concepts Experience testing LLM-integrated features or AI-driven workflows Familiarity with ML-related tooling Experience with API testing, JSON validation, and test data generation Ability to test nondeterministic systems Strong analytical, troubleshooting, and communication skills Experience testing cloud-integrated systems, preferably on AWS Bachelor’s degree in Computer Science, Information Systems, or equivalent practical experience
Responsibilities:
Design and execute test plans for A.I and Machine Learning driven features Develop Python-based automation and evaluation scripts Test LLM-based features for accuracy and guardrail effectiveness Validate data pipelines and model integration points Conduct regression testing on ML models and AI workflows Test enterprise-scale behavior analysis features Validate security, privacy, and zero-trust requirements for AI model interactions Collaborate with ML engineers and cross-functional teams Analyze logs, metrics, and model outputs to detect anomalies
