Langsung ke konten
Global sound
KubeFM

KubeFM

1 Million Tokens Per Second on Kubernetes, with Federico Iezzi

· 48 mnt · Musim 9 Episode 1

Catatan episode

GPU inference throughput depends on more than accelerator generation or count.
Memory bandwidth, model parallelism, cache configuration, and the load generator itself all influence measured throughput.
Federico Iezzi, Customer Engineer at Google Cloud, explains how his team achieved 1 million output tokens per second using Qwen 3.5 27B, vLLM, GKE Autopilot, and NVIDIA B200 GPUs.
The discussion covers:

• Why memory bandwidth limits decode performance

• How Federico chose between tensor and data parallelism

• What changed after enabling multi-token prediction and reducing the KV cache…

Situs: ku.bz

Live, satu jam terakhir: 5.583 lagu di 2.386 stasiun di 114 negara

Jelajahi dunia lewat radio

Cara lain untuk menjelajah (7)

Dengar sesuatu yang bagus? Simpan, atau langsung buka di Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
City FM 99.1 ShandongLanjut mendengarkan