Machine Learning Infrastructure Engineer
True Talent Solutions Partners · San Francisco · onsite
ABOUT THE ROLE
Join an AI team developing foundation models for physical systems, with weather as an initial application. You will build the distributed infrastructure that supports large-scale model training and inference, helping advance AI for science and physical AI.
WHAT YOU'LL DO
- Design, deploy, and maintain distributed machine learning training and inference clusters.
- Build scalable end-to-end pipelines for petabyte-scale datasets and the full model lifecycle.
- Research and evaluate training approaches, including parallelization strategies and numerical precision trade-offs across model scales.
- Profile and debug low-level GPU operations to improve performance.
- Analyze experiment results, including through ablation studies, and apply relevant research to infrastructure work.
WHAT WE'RE LOOKING FOR
- Two to ten years of experience building large-scale machine learning infrastructure for pre-trained or foundation models.
- Hands-on experience across compute clusters, data infrastructure, distributed training, and inference, with a record of optimizing large-scale workloads.
- Proficiency with distributed training frameworks such as FSDP or DeepSpeed, and experience with GPU performance optimization and debugging, including CUDA and JAX.
- Knowledge of cloud platforms such as GCP, AWS, or Azure, plus containers and orchestration tools such as Docker and Kubernetes.
- Familiarity with model serving, distributed task management, and ML system monitoring, logging, observability, and version control.
- Experience in AI for science or physical AI, such as robotics, autonomous systems, or biology, and a generalist approach across the ML lifecycle.
- Comfort working in fast-paced, time-sensitive environments and learning unfamiliar domains.
COMPENSATION & BENEFITS
Base salary: $200,000 to $400,000 annually. Visa sponsorship is not available.
LOCATION
On-site in San Francisco, California, United States, five days per week. Relocation support is provided.