Senior AI Infrastructure and Optimization Engineer
vantaca · Palo Alto, United States
HOAi is the fastest-growing company revolutionizing the community association management industry. Our AI workforce platform integrates machine learning technology to streamline labor-heavy processes, eliminating inefficiencies and driving scalability. With rapid growth in the AI space, we are pushing boundaries to redefine industry standards. HOAi Agents go beyond traditional AI by proactively executing complex, multi-step processes with human-like reasoning—working autonomously, 24/7, across your entire operation. This transformation optimizes labor costs, enables growth without additional hires, and ensures faster, higher-quality service for residents and board members.
HOAi was acquired by Vantaca in the fall of 2024. Vantaca just achieved unicorn status with a $1.25B valuation, so it's safe to say we're past the "scrappy startup phase." We're not just building a successful company – we're building the category-defining platform that will transform how an entire industry operates. Here's the reality of our trajectory:- Growing 100% year-over-year
- Our AI product (HOAi) went from $0 to millions in months
- Backed by Cove Hill Partners and JMI Private Equity
- 6M+ doors on our platform, displacing legacy systems
About the Role
We run millions of AI tasks in production and are hiring a Senior AI Infrastructure and Optimization Engineer to own the systems, experiments, and feedback loops that determine how those tasks use AI models.
AI models are improving rapidly, but model capability alone does not determine production performance. Model choice, prompts, context, tool use, routing, retries, and execution design all affect whether a task succeeds and what that success costs.
Your mandate is to maximize successful task completion per dollar while meeting our quality, reliability, and latency requirements. You will build evaluation and observability infrastructure, benchmark models and configurations, analyze production behavior, and translate findings into production improvements. You will also design optimization loops that search for better prompts, model configurations, routing policies, agent strategies, and execution plans.
This is an engineering-heavy, highly empirical role. You will work across model providers, prompts, agent architectures, routing strategies, production traces, and optimization systems to determine what works at scale for our workloads. TypeScript is the primary implementation language.
What You Will Own
Production AI Infrastructure
- Build the infrastructure required to measure, operate, and improve millions of AI tasks reliably.
- Build TypeScript services and libraries for model execution, routing, retries, fallbacks, and structured outputs.
- Instrument AI workflows for quality, latency, reliability, token usage, and cost.
- Create consistent interfaces across model providers and configurations.
- Provide end-to-end traceability across prompts, models, tools, agent steps, retries, and outcomes.
- Define durable schemas for tasks, evaluations, experiments, model versions, and production results.
- Design systems for safe rollout, rollback, and comparison of model and workflow changes.
- Improve the reliability and operational efficiency of high-volume AI workloads.
- Partner with platform and product engineers so teams can adopt optimization capabilities throughout the stack.
- This is a production engineering role. The systems you build will run in live, high-volume production environments.
Model Evaluation
- Build and maintain evaluation suites that represent our production workloads and predict real production performance.
- Define task-specific quality metrics and acceptance thresholds.
- Build golden datasets from production workloads, human-reviewed examples, edge cases, and synthetic scenarios.
- Develop deterministic checks, automated graders, and LLM-as-judge evaluations where each method is appropriate.
- Calibrate automated evaluations against human judgment.
- Build regression tests that detect degradation as models, prompts, tools, and agent architectures change.
- Sample and analyze production tasks to identify new failure modes.
- Continuously update evaluations as production behavior changes.
- Connect offline evaluation results to online task, user, and business outcomes.
Model Experimentation
- Continuously evaluate the model ecosystem and the configurations around it, including: Models from OpenAI, Anthropic, Google, open-weight providers, and emerging vendors.
- Model sizes, capability tiers, reasoning effort, and inference-time compute.
- Prompt, system-instruction, context-selection, and context-compression strategies.
- Tool-use configurations, structured outputs, and agent architectures.
- Retries, fallbacks, escalation policies, and model cascades.
- Execution parallelism, task decomposition, and planning approaches.
- Experiments should be reproducible, statistically sound, and connected to production outcomes. They should produce clear recommendations and deployable changes rather than one-off benchmark results.
Continuous Optimization
- Build and operate systems that continuously search for better AI configurations across prompts, model selection, routing, reasoning budgets, context construction, tool use, agent workflows, retries, caching, parallelism, and workflow-specific policies.
- Select the appropriate search method for each problem, from manual experimentation and grid or random search to Bayesian optimization, bandit methods, evolutionary approaches, and production feedback loops.
- Optimization must be safe and measurable. Build guardrails against flawed graders, narrow-set overfitting, segment-level regressions, reward hacking, and hidden operational costs.
- Candidate generation and evaluation pipelines.
- Experiment queues and prioritization.
- Champion and challenger comparisons.
- Automatic promotion and rollback criteria.
- Exploration and exploitation policies.
- Segment-level monitoring and regression detection.
- Human-review escalation for ambiguous or high-impact changes.
- Versioning for prompts, models, tools, datasets, graders, and optimization policies.
Cost and Performance Optimization
- Establish the Pareto frontier across quality, cost, latency, and reliability for each major workload. Investigate opportunities such as: Use smaller models where larger models provide little incremental value.
- Route requests dynamically based on task difficulty.
- Reduce unnecessary context, output tokens, agent steps, tool calls, retries, and model calls.
- Tune reasoning and token budgets.
- Compress prompts and context without losing required information.
- Use caching, reuse, model cascades, fallbacks, and execution parallelism where they improve economics.
- Compare fine-tuning or distillation with frontier-model APIs where appropriate.
- Improve provider selection and workload placement.
- Reduce operational overhead in high-volume execution paths.
- The objective is to minimize cost per successful task while preserving the quality and reliability users require.
Production Deployment and Monitoring
- Turn experimental results into measurable improvements in production.
- Build dashboards for quality, success rate, latency, reliability, and AI economics by workflow, model, configuration, and customer segment.
- Run controlled production experiments and A/B tests.
- Roll out model migrations and configuration changes safely.
- Detect regressions after model or provider updates.
- Establish automated gates for model, prompt, tool, and agent changes.
- Monitor optimization systems for reward hacking and grader exploitation.
- Identify where production behavior diverges from offline evaluation.
- Partner with product and engineering teams to determine where additional model capability creates meaningful value.
- Quantify optimization results in successful tasks, user outcomes, and spend.
What Success Looks Like
Within your first several months, the team should be able to