Research Engineer, Evals
True Talent Solutions Partners · San Francisco · onsite
ABOUT THE ROLE
Build evaluation systems for a web data extraction product, defining and measuring output quality across diverse websites, formats, and edge cases. You will own the process from evaluation methodology and datasets through to insights that inform model and product decisions.
WHAT YOU'LL DO
- Design metrics that define high-quality structured extraction across varied web pages and data formats.
- Build scalable pipelines and evaluation harnesses to measure output quality rigorously.
- Generate and curate trustworthy datasets with reference extractions and ground-truth labels.
- Connect evaluation results to model and product decisions, closing the feedback loop.
- Make evaluation findings actionable for the broader team.
WHAT WE'RE LOOKING FOR
- Four or more years of relevant experience, including at least two years working on evaluation systems.
- Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, evaluation frameworks, and statistical analysis.
- Experience designing and building evaluations for LLM-based structured extraction from messy, real-world web pages or documents, including field-level accuracy measurement at scale.
- Background building or evaluating large-scale web or document extraction systems and creating ground-truth datasets.
- Statistical rigor in metric design and interpretation, including sampling, significance, estimator bias and variance, error bounds, and judge-to-human agreement.
- Ability to combine research methodology with hands-on engineering and use evaluation results to guide real model or product changes.
COMPENSATION & BENEFITS
Base salary range: $250,000 to $290,000 USD annually. Visa sponsorship is available.
LOCATION
Hybrid, based in San Francisco, California. The role requires working on-site five days per week in San Francisco or Toronto.