Zum Inhalt springen
Global sound
The AI Kubernetes Show

The AI Kubernetes Show

Disaggregated Inference on Kubernetes

· 57 Min. · Staffel 2 Folge 8

Shownotes

Every LLM inference request splits into two phases with completely different hardware profiles: pre-fill and decode. But most Kubernetes clusters still run them on the same pod. Morgan Foster, who works on the llm-d project and the AI Gateway Working Group out of Red Hat's Office of the CTO, joins William Morgan to explain what that means.
Foster and Morgan get into why real-time inference workloads fit the primitives Kubernetes already has, while training often runs on Slurm instead. They walk through disaggregated inference: how pre-fill and decode get scheduled on separate pods, how KV…

Website: podcasters.spotify.com

Live, letzte Stunde: 7.415 Songs auf 3.005 Sendern in 129 Ländern

Entdecke die Welt über das Radio

Mehr zum Entdecken (9)

Etwas Schönes gehört? Speicher es oder öffne es direkt in Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Radio Candela ClassicsWeiterhören