AI/ML Evaluation Engineer - Global Solutions Provider (Brazil)

Advanced Python skills for writing, debugging, and automating scripts. Strong SQL proficiency for manipulating large datasets. Hands-on experience with Python libraries like Pandas and NumPy. Ability to clean, standardize, and analyze structured and unstructured data. Experience inspecting datasets, visualizing distributions, and preparing data. Solid understanding of large language models, prompt behavior, and grounding concepts. Knowledge of retrieval-augmented generation (RAG) flows and embedding-based search. Awareness of vector similarity concepts like cosine similarity and dot product. Experience with at least one LLM evaluation framework (RAGAS, TruLens, LangSmith, etc.). Ability to design or implement custom LLM-as-Judge evaluation systems. Applied understanding of statistical concepts like variance, confidence intervals, precision/recall, and correlation. Ability to translate ambiguous quality expectations into measurable metrics. Familiarity with cloud-run services and automation pipelines, preferably on GCP. Ability to learn new infrastructure tools quickly. Strong analytical and problem-solving abilities. Excellent communication skills for cross-functional collaboration.

Write Python and SQL scripts to evaluate LLM outputs. Design and implement LLM-as-Judge evaluations with scoring rubrics. Define and calculate metrics such as exact match, F1, ROUGE, and cosine similarity. Build and maintain ground-truth datasets for benchmarking. Automate evaluation workflows and integrate them into CI/CD pipelines. Analyze large unstructured datasets for inconsistencies and biases. Diagnose failure modes such as hallucinations and irrelevant answers. Produce reports summarizing evaluation findings and quality trends. Collaborate with cross-functional teams to define quality standards. Document all processes, evaluation setups, and architecture diagrams. Maintain reproducibility and traceability for evaluation runs and datasets.