Research Engineer, Evals
True Talent Solutions Partners · San Francisco · onsite
ABOUT THE ROLE
Build evaluation systems for a web data extraction product, defining how output quality is measured across diverse websites, formats, and edge cases. You will combine research and engineering to turn evaluation results into actionable model and product improvements.
WHAT YOU'LL DO
- Design rigorous metrics for evaluating structured extraction quality across varied web content.
- Build scalable evaluation pipelines, testing harnesses, and supporting data systems.
- Generate and curate ground-truth datasets for trustworthy measurement.
- Connect evaluation findings to model and product decisions, closing the quality feedback loop.
- Make evaluation results clear and useful to the broader team.
WHAT WE'RE LOOKING FOR
- Typically 4 to 10 or more years of relevant experience, including at least 2 years working on evaluations.
- Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, evaluation frameworks, and statistical analysis.
- Experience building evaluation systems for LLM-based structured extraction from messy, real-world web pages or documents, including field-level accuracy measurement at scale.
- Experience designing evaluation methods and metrics, building ground-truth datasets, and applying statistical rigor, including sampling, estimator bias and variance, error bounds, and judge-to-human agreement.
- Ability to combine research methodology with engineering implementation and use evaluation results to guide real model or product decisions.
COMPENSATION & BENEFITS
Base salary: $250,000 to $290,000 USD annually. Visa sponsorship is available.
LOCATION
Hybrid, based in San Francisco, California. The role requires working on-site five days a week in San Francisco or Toronto, with relocation before starting if needed.