Software Engineer, RL Environments
True Talent Solutions Partners · San Francisco · onsite
ABOUT THE ROLE
Design datasets, evaluation rubrics, and reward signals that help improve frontier AI models. You will work with research teams to test data-collection strategies, investigate model failure modes, and deliver evaluations and training data that inform model development.
WHAT YOU'LL DO
- Design data slices that expose meaningful model failure modes across areas such as finance, software, and enterprise workflows.
- Build and refine evaluation rubrics and reward signals for RLHF and RLVR training pipelines.
- Model annotator behavior and run experiments to improve model capabilities.
- Develop quantitative methods for assessing dataset quality, diversity, and downstream impact.
- Create and manage real-world and synthetic data pipelines.
- Translate research objectives into concrete data and evaluation specifications in collaboration with research teams.
WHAT WE'RE LOOKING FOR
- One to four years of hands-on experience designing machine learning data-collection strategies and evaluation rubrics.
- Strong familiarity with LLM training and evaluation methods, including experience creating data slices to reveal model failure modes.
- Ability to design lightweight experiments, move quickly, and extract useful insights from messy results.
- Comfort working across domains such as finance, software engineering, and policy.
- A practical, builder-oriented approach and genuine interest in how data structure and quality shape model behavior.
COMPENSATION & BENEFITS
Base salary: $260,000 to $290,000 annually. Visa sponsorship is available.
LOCATION
On-site in San Francisco, California.