Ir para o conteúdo
Global sound
Software Engineering Radio - the podcast for professional software developers

Software Engineering Radio - the podcast for professional software developers

SE Radio 747: Sabina Anja on How AI Workloads Shape GPU Network Design

· 54 min · Episódio 747

Notas do episódio

Sabina Anja, Chief Technologist and executive advisor at VMware Cloud Foundation, talks with host Amey Ambade about how AI workloads reshape GPU network design, and why the network moves into the critical path of computation once a model is too large for a single GPU. Using Llama 3's 405 billion parameters as a starting point, Sabina explains how sharding spreads model weights across many GPUs. She then walks through the difference between scaling up inside a server over NVLink and scaling out across servers over the cluster network. She describes collective traffic as synchronous: no GPU moves to the next step until every GPU finishes, so one slow GPU or congested link, called a straggler, stalls the whole job. The conversation covers data, tensor, and pipeline parallelism and the distinct traffic pattern each produces, with analogies such as splitting a restaurant bill to explain all-reduce. It then moves to physical topology, including rail-optimized designs, dedicated back-end AI networks that echo the storage area networks of the past, and why oversubscription, a sensible cost bet in enterprise networks, fails when AI traffic runs every link at full capacity.
The discussion then turns to congestion and latency: why synchronized collectives produce sudden bursts that overflow switch buffers, how older techniques such as priority flow control return to pause traffic before loss, and how overlapping communication with computation hides some network delay. On inference, Sabina explains how sharded models pay network latency on every generated token, why the prefill and decode phases have different resource demands, and why some systems separate them into different GPU pools and move the KV cache between them. She also describes how mixture-of-experts models can concentrate traffic on the servers that host popular experts. The episode closes with a vendor-neutral framework for evaluating network fabrics: treat the network as a design constraint in the same way as compute and memory, and give the network team concrete information about the traffic a workload will generate. Her summary of the challenge is that AI networking is a watermelon-through-a-keyhole problem.

Site: se-radio.net

Ao vivo, última hora: 6.990 músicas em 2.661 rádios de 114 países

Descubra o mundo pelo rádio

Mais formas de descobrir (7)

Ouviu algo legal? Salve ou abra direto no Spotify.

★ Artista da semana A maior subida na rádio esta semana
A tua música aqui? Desde € 9 por semana →
New Radio Oriental Fm 98.3Continuar ouvindo