Hiring: AI Engineer | Agentic AI, AWS Bedrock AgentCore & Langfuse | 15+ Years
TMS LLC · Los Angeles, California, United States · hybrid
Job Title: AI Engineer / Generative AI Engineer
Location: Los Angeles, CA (Hybrid)
Duration: 3+ Months (Extension Possible)
Experience: 15+ Years
Job Summary
We are seeking a hands-on AI Engineer with expertise in Python, Agentic AI, AWS Bedrock AgentCore, and LLM evaluation to build, deploy, monitor, and continuously improve production-grade AI agents.
Key Responsibilities
- Develop AI agents using Python and Claude Agent SDK.
- Deploy agents using AWS Bedrock AgentCore Runtime.
- Design golden datasets and implement LLM-as-a-Judge evaluations.
- Integrate automated evaluations into CI/CD pipelines as quality gates.
- Implement Langfuse/OpenTelemetry for AI observability and tracing.
- Evaluate agent accuracy, task completion, tool execution, and performance.
- Support RAG pipelines and AWS data processing workflows.
Required Skills
- Python and Agentic AI development.
- AWS Bedrock AgentCore Evaluations / Bedrock evaluation.
- Golden-dataset design and curation.
- LLM-as-a-Judge design and calibration against human labels.
- Automated evaluation and CI gate integration.
- Langfuse and/or OpenTelemetry.
Good to Have
- RAGAS or equivalent evaluation frameworks.
- Trajectory evaluation and agentic red-teaming.
- Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.
Qualifications
- 6–8+ years of software engineering, cloud engineering, or AI/ML experience.
- Hands-on experience with production AI agents and continuous evaluation.
- Strong problem-solving and troubleshooting skills.
All your information will be kept confidential according to EEO guidelines.