Senior Evaluation Algorithm Engineer
New
B
BinanceAI evaluation
Location: Australia; Hong Kong; TaiwanFull-TimeSenior
Salary not disclosed
Apply NowOpens the employer's application page
Job Details
- Required Skills
- Python
Requirements
- Have a master's degree or above in Computer Science, Artificial Intelligence, Mathematics, Statistics, or a related field.
- Have a solid algorithmic foundation and understand LLM principles, training, and fine-tuning processes.
- Have hands-on LLM evaluation experience at a large tech company, including evaluation for commercial deployment.
- Be familiar with the evaluation pipeline from data preparation and rubric design to evaluation-driven R&D.
- Be familiar with human evaluation, model-based automatic evaluation / LLM-as-a-judge, and metric computation, including their applicable boundaries.
- Be able to define evaluation dimensions for different business scenarios and write clear, actionable, discriminative rubrics.
- Have systematic control over evaluation data representativeness, annotation consistency, and result reliability.
- Be proficient in Python and experienced in evaluation workflow automation, benchmark construction, or evaluation platform development.
- Be able to handle data processing, evaluation script writing, and result analysis independently.
- Understand business needs and communicate findings as improvement directions, collaborating across teams.
- Experience with dialogue systems, AI agents, or financial/trading LLMs is a bonus.
- Experience building AI training/evaluation data or annotation systems is a bonus.
- Familiarity with RLHF, reward models, or preference data is a bonus.
Responsibilities
- Design end-to-end LLM evaluation plans for dialogue, financial trading, and other business scenarios.
- Build evaluation metric systems and rubrics that produce quantifiable, reproducible, and explainable conclusions.
- Design evaluation datasets, define dimensions and scenario coverage, and establish annotation guidelines and quality controls.
- Build benchmarks that reflect business needs and distinguish model performance.
- Analyze model capability boundaries and failure modes, and produce actionable improvement recommendations.
- Collaborate with algorithm and product teams to drive model iteration.
- Build evaluation platforms and toolchains to automate and scale evaluation workflows.
- Work with algorithm, product, and data teams to translate objectives into evaluation standards and implement findings.
View Full Description & ApplyYou'll be redirected to the employer's site
