Note dell’episodio
This episode examines why speculative decoding, long thought to give little benefit for sparse mixture-of-experts models at batch size one, actually helps most in that regime for many newer models. It covers ten MoE models from 30 billion to a trillion parameters, and explains how a drafter proposes tokens that the target model verifies in a single pass. It then contrasts coarse-grained experts (Mixtral-style) with fine-grained ones (Qwen3, Kimi-K2.5), where each token touches only a tiny slice of the layer. The discussion challenges the bandwidth-bound cost model from prior work, which predicts a speedup curve that rises, peaks at mid-sized batches, then falls. The authors point to two missing factors: adjacent tokens routing to overlapping experts, and real GPU execution dynamics. The episode also explains target efficiency, a metric that isolates verification cost from draft acceptance, and outlines the vLLM benchmarking setup on H100 and B200 GPUs. Listeners who care about low-latency serving for agents, test-time scaling, or RL rollouts will find it useful for deciding when speculative decoding is worth enabling.
Sources:
1. Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models — Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli, 2026
http://arxiv.org/abs/2610.38981
2. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2022 (arXiv); ICML 2023
https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding
3. Accelerating Large Language Model Decoding with Speculative Sampling — Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper, 2023
https://scholar.google.com/scholar?q=Accelerating+Large+Language+Model+Decoding+with+Speculative+Sampling
4. MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE — Zongle Huang, Lei Zhu, Zongyuan Zhan, Ting Hu, Weikai Mao, Xianzhi Yu, Yongpan Liu, Tianyu Zhang (verify author list), 2025
https://scholar.google.com/scholar?q=MoESD%3A+Unveil+Speculative+Decoding%27s+Potential+for+Accelerating+Sparse+MoE
5. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2025
https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test
6. Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference — Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. Panda, 2024
https://scholar.google.com/scholar?q=Exploiting+Inter-Layer+Expert+Affinity+for+Accelerating+Mixture-of-Experts+Model+Inference
7. Fast Inference of Mixture-of-Experts Language Models with Offloading — Artyom Eliseev, Denis Mazur, 2023
https://scholar.google.com/scholar?q=Fast+Inference+of+Mixture-of-Experts+Language+Models+with+Offloading
8. DeepSeekMoE: Towards Ultimate…
