Machine Learning Infrastructure Engineer
True Talent Solutions Partners · San Francisco · onsite
ABOUT THE ROLE
Join an AI team building foundation models for physical systems, starting with weather. You will develop the distributed infrastructure that enables large-scale model training and inference, helping advance AI for scientific and real-world applications.
WHAT YOU'LL DO
- Design, deploy, and maintain distributed machine learning training and inference clusters.
- Build scalable end-to-end pipelines for petabyte-scale datasets and model training across the machine learning lifecycle.
- Research and evaluate training approaches, including parallelization strategies and numerical precision trade-offs across model scales.
- Profile and debug low-level GPU operations to improve performance.
- Track relevant research and apply promising ideas to infrastructure and model development.
WHAT WE'RE LOOKING FOR
- Two to ten years of experience building large-scale machine learning infrastructure for pretrained foundation models.
- Experience with distributed training frameworks such as FSDP or DeepSpeed, and with optimizing large-scale training, data, or inference workloads.
- Strong background in distributed compute, data infrastructure, inference systems, and model serving.
- Familiarity with GPU performance optimization and debugging, including CUDA and JAX, and with Python, C++, Linux, and NVIDIA hardware.
- Knowledge of cloud platforms such as GCP, AWS, or Azure, and containerization and orchestration tools such as Docker and Kubernetes.
- Experience in physical AI or AI for science, such as robotics, autonomous systems, or biology, and motivation to work on scientific or physical-world applications.
- Comfort working across the machine learning lifecycle in fast-paced, time-sensitive environments; experience analyzing training experiments and ablation studies is valuable.
COMPENSATION & BENEFITS
Base salary: $200,000 to $400,000 annually. Visa sponsorship is not available.
LOCATION
On-site, five days per week in San Francisco, California. Relocation support is provided.