Maximizing GPU Utilization: Heterogeneous Pipelines with Ray and Kubernetes

Summary
In this episode Robert Nishihara, co-founder of Anyscale and co-creator of Ray, talks about maximizing hardware utilization for AI and data-intensive workloads. He explores Ray’s evolution alongside Kubernetes and PyTorch, and why consolidation at these layers has enabled a new generation of complex, heterogeneous workloads. Robert explains how data preparation has shifted to GPU- and inference-heavy, multimodal pipelines; where Ray fits compared to Spark and workflow orchestrators; and why Ray excels at composing heterogeneous pools of compute, handling failures, and scaling complex systems like multi-node LLM inference and reinforcement learning. He digs into practical strategies for boosting GPU utilization across training and inference, elasticity and prioritization of workloads, topology-aware scheduling, and the importance of fast failure recovery as hardware scales from nodes to racks. If you’re wrestling with expensive GPUs, multimodal data curation, or cross-node LLM inference, this conversation offers concrete mental models and architectural guidance.

Announcements

• Hello and welcome to the Data Engineering Podcast, the show about modern data management
• Your host is Tobias Macey and today I'm interviewing Robert Nishihara about the challenges of maximizing the utility of your available hardware for AI applications

Interview

• Introduction
• How did you get involved in the area of data management?
• Can you start by giving an overview of the major contributors to wasted or idle compute?
• Why does it matter if the available compute isn't being maximized?
• What are some of the typical ad-hoc methods that teams might use to try to get the most out of their available hardware (especially GPUs)? 
• What are the most interesting, innovative, or unexpected ways that you have seen Ray used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on Ray and distributed compute for data and AI?
• When is Ray the wrong choice?
• What do you have planned for the future of Ray?

Contact Info

• LinkedIn

Parting Question

• From your perspective, what is the biggest gap in the tooling or technology for data management today?

Closing Announcements

• Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.

Links

• AnyScale
• Ray
• Deep Learning
• Computer Vision
• Kubernetes
• Cursor
• Claude Code
• Kube-Ray
• PyTorch
• Tensorflow
• Theano
• Caffe
• vLLM
• SGLang
• Ray Tune
• Neural Network
• Learning Rates
• Reinforcement Learning
• AlphaGo
• Cursor Composer 2
• ImageNet
• Transformer Architecture
• Stochastic Gradient Descent
• Airflow
• Dagster
• Flyte
• Mixture of Experts
• Prefill
• Temporal
• Actor Framework
• RDMA == Remote Direct Memory Access
• Neoclouds
• AI Engineering Podcast Episode

The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA