Research Engineer, Evals
True Talent Solutions Partners · San Francisco · onsite
ABOUT THE ROLE
Build evaluation systems for a web data extraction product that turns pages into structured, LLM-ready information. You will shape how quality is measured, create the infrastructure to measure it at scale, and connect evaluation results to model and product decisions.
WHAT YOU'LL DO
- Design metrics for extraction quality across diverse websites, formats, and edge cases.
- Build scalable pipelines and evaluation harnesses to measure output quality rigorously.
- Generate and curate ground-truth datasets for trustworthy evaluation.
- Close the feedback loop by translating evaluation results into model and product improvements.
- Make quality results clear and actionable for the broader team.
WHAT WE'RE LOOKING FOR
- Four or more years of relevant engineering experience, including at least two years focused on evaluation work.
- Experience building evaluations for LLM extraction of unstructured data, measuring whether fields are correctly extracted from messy, real-world pages or documents.
- Experience building or evaluating large-scale web or document extraction systems and developing evaluation metrics and systems from the ground up.
- Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, and evaluation frameworks.
- Statistical rigor in metric design and interpretation, including sampling, significance, estimator bias and variance, error bounds, and judge-to-human agreement.
- Experience creating ground-truth datasets and using evaluation findings to inform real model or product decisions.
- A combination of research methodology and hands-on engineering, plus comfort working through ambiguous problems.
COMPENSATION & BENEFITS
Base salary: $250,000 to $290,000 USD annually. Visa sponsorship is available.
LOCATION
Hybrid, based in San Francisco, California, or Toronto, Canada. The role requires working on-site five days a week.