About the Company Our client is an early-stage AI company building generative models that learn the physics and geometry of the real world in 3D, rather than working from flat images. The work is aimed at use cases in robotics, AR/VR, gaming, and film.
The team is small, and researchers and engineers work closely together on both the models and the systems that take them into products.
About the Role This role owns the infrastructure behind the company’s training and evaluation of large diffusion-based generative models. That covers GPU clusters, storage, orchestration, and the path to model serving.
It is hands-on and systems-focused. You will work directly with the researchers who use what you build, so priorities come from real training problems rather than a roadmap handed down to you.
What You’ll Do
- Own and evolve the ML and cloud infrastructure used to train and evaluate large models
- Provision, scale, and maintain multi-node, multi-GPU environments on cloud and/or on-prem, including scheduling, quotas, and capacity planning
- Support distributed training (PyTorch DDP/FSDP, NCCL) and keep large runs stable, performant, and reproducible
- Build and optimize storage and networking for petabyte-scale datasets
- Deploy workloads with Docker and Kubernetes, and manage infrastructure-as-code with Terraform
- Set up monitoring, logging, and alerting for cluster health, job performance, and cost, and define SLOs and incident response practices
- Manage secrets, IAM, and network security across research and production
- Work with researchers to unblock training and support the move from research to served models
- 3+ years in infrastructure, platform, or cloud engineering
- Hands-on GPU performance debugging (CUDA/NCCL concepts, utilization, network bottlenecks, profiling)
- Strong experience running AWS, GCP, or Azure, including networking, IAM, and cost management
- Proficiency with Docker, Kubernetes, and Terraform
- Strong scripting and automation skills in Python plus Bash or PowerShell
- Familiarity with distributed training in PyTorch (DDP/FSDP or similar)
- Experience with observability tooling such as Prometheus/Grafana, OpenTelemetry, or ELK
- Experience building CI/CD for infrastructure and ML workflows
- Prior ML infrastructure experience supporting large-scale training
- Experience running on-prem GPU clusters alongside cloud capacity
- Experience with model evaluation and serving infrastructure
- Experience running on-call and incident response
- Ownership. You will be responsible for the infrastructure behind a research program, not one corner of a larger platform.
- Scale. Multi-node GPU training and petabyte-scale datasets are part of the daily work.
- Proximity to research. You will work alongside the people who depend on your systems.
- Influence. In a small team, your views on tooling and architecture carry weight.
If this sounds relevant, get in touch for a confidential conversation.
