Founding AI Quality & Release Engineer
True Talent Solutions Partners · New York · onsite
ABOUT THE ROLE
This is a founding-level software engineering role focused on building reliable, AI-powered verification and release infrastructure. You will develop the systems that evaluate agent-based products, strengthen CI signals, and help ensure safe, predictable releases.
WHAT YOU'LL DO
- Build and maintain test harnesses that use agents to verify product behavior.
- Develop evaluation logic, including LLM graders and regression coverage for agentic systems.
- Create automated release gates that make readiness visible and prevent unsafe releases.
- Own CI health, release pipelines, artifact publication, and deployment automation.
- Improve the reliability of staging and demonstration environments.
- Investigate quality gaps and incidents, and improve systems to prevent recurring issues.
WHAT WE'RE LOOKING FOR
- Seven to fifteen years of experience in quality engineering, release engineering, or platform engineering.
- Recent hands-on experience using AI-native methods to test or evaluate LLM-based or agentic systems.
- Practical experience with LLM APIs, tool calling or MCP, and evaluation methods.
- Strong familiarity with GitHub Actions and GitHub-based CI/CD, including container builds and pipeline automation.
- Experience with Python, TypeScript, Docker, Kubernetes, and Helm.
- Working knowledge of at least one major cloud or data platform, such as AWS, Azure, Databricks, or Snowflake.
- A track record of taking independent ownership of engineering systems, ideally in a high-velocity startup or while modernizing established engineering practices.
- Authorization to work in the United States or Canada without visa sponsorship.
COMPENSATION & BENEFITS
Base salary: $170,000 to $200,000 USD annually. Visa sponsorship is not available.
LOCATION
Remote within the United States or Canada. Primary location: New York, New York, United States.