AI Query Evaluation Specialist (Copilot Competitive Intelligence)
New
B
Blueprint TechnologiesTechnology Solutions
Remote (U.S.-based)Full-TimeMiddle
Salary80,000 - 95,000 USD per year
Apply NowOpens the employer's application page
Job Details
- Languages
- English, Japanese, Korean, French, Chinese, German, Italian
- Required Skills
- Microsoft Excel
Requirements
- Strong English reading comprehension with the ability to interpret subtle differences in user intent
- Demonstrated analytical thinking and logical reasoning skills
- Experience working with structured data or annotation workflows
- Familiarity with Microsoft Excel or similar data analysis tools
- Strong user empathy and understanding of how diverse users formulate queries
- Curiosity and familiarity with modern AI tools (e.g., Copilot, ChatGPT, Gemini)
- High attention to detail with a track record of delivering consistent, high-quality work
- Reliable, proactive, and adaptable in fast-changing environments
- Prior experience in data annotation, content evaluation, or dataset curation for AI or search products (Preferred)
- Experience with AI evaluation, search relevance, or linguistic analysis (Preferred)
- Basic statistical or data analysis knowledge (Preferred)
- Demonstrated ability to quickly learn and interpret unfamiliar domains (Preferred)
- Fluency in English plus at least one additional language: Japanese, Korean, French, Chinese, German, or Italian (Preferred)
Responsibilities
- Support an initiative focused on evaluating and improving AI-powered search and assistant experiences
- Work with real user queries to build and refine evaluation datasets
- Benchmark Microsoft Copilot against leading AI systems
- Review and analyze real user query logs to identify queries with clear intent and representativeness
- Curate and maintain high-quality datasets for AI system evaluation
- Annotate queries across multiple evaluation dimensions (web search need, PII presence, domain expertise requirement, AI response quality attributes)
- Ensure annotations are consistent, structured, and aligned with evaluation guidelines
- Use tools such as Excel to organize, review, and summarize evaluation outputs
- Identify trends in query patterns and provide feedback to improve dataset coverage and quality
- Maintain high attention to detail, documentation quality, and evaluation integrity
View Full Description & ApplyYou'll be redirected to the employer's site
