Avsnittsbeskrivning
This episode examines "Nemotron-Labs-Diffusion," an 8B model that runs in three modes from one set of weights: ordinary autoregressive decoding, block-diffusion parallel decoding, and self-speculation. In self-speculation, the diffusion pathway drafts tokens and the causal pathway verifies them, so no separate drafter is needed. It first explains why single-user decoding is memory-bound, then covers the prior approaches: masked and block diffusion, speculative decoding, and multi-token-prediction and Eagle3 drafters. A central theme is that the headline claims mix different metrics. Tokens per forward pass, acceptance rate and real hardware throughput are not interchangeable. A self-speculation cycle costs two forward passes, so true tokens per pass is roughly half the raw acceptance count. The episode also covers the training recipe, which uses a noisy block-bidirectional stream alongside a strictly causal clean stream. It notes that the "speed-of-light" analysis is an oracle upper bound, not a deployable sampler. Listeners get a clear framework for reading speedup claims in LLM inference papers critically.
Sources:
1. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding — Yonggan Fu, Lexington Whalen, Abhinav Garg, Chengyue Wu, Maksim Khadkevich, Nicolai Oswald, Enze Xie, Daniel Egert, Sharath Turuvekere Sreenivas, Shizhe Diao, Chenhan Yu, Ye Yu, Weijia Chen, Sajad Norouzi, Jingyu Liu, Shiyi Lan, Ligeng Zhu, Jin Wang, Jindong Jiang, Morteza Mardani, Mehran Maghoumi, Song Han, Ante Jukić, Nima Tajbakhsh, Jan Kautz, Pavlo Molchanov, 2026
http://arxiv.org/abs/2607.05722
2. Fast Inference from Transformers via Speculative Decoding — Yaniv Leviathan, Matan Kalman, Yossi Matias, 2022 (ICML 2023)
https://scholar.google.com/scholar?q=Fast+Inference+from+Transformers+via+Speculative+Decoding
3. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (and the earlier EAGLE / EAGLE-2) — Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang, 2024-2025
https://scholar.google.com/scholar?q=EAGLE-3%3A+Scaling+up+Inference+Acceleration+of+Large+Language+Models+via+Training-Time+Test+%28and+the+earlier+EAGLE+%2F+EAGLE-2%29
4. Large Language Diffusion Models (LLaDA) — Shen Nie, Fengqi Zhu, et al., 2025
https://scholar.google.com/scholar?q=Large+Language+Diffusion+Models+%28LLaDA%29
5. Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (BD3-LMs) — Marianne Arriola, Subham Sekhar Sahoo, et al., 2025 (ICLR 2025)
https://scholar.google.com/scholar?q=Block+Diffusion%3A+Interpolating+Between+Autoregressive+and+Diffusion+Language+Models+%28BD3-LMs%29
6. Better & Faster Large Language Models via Multi-token Prediction — Fabian Gloeckle, Badr Youbi Idrissi, et al. (Meta FAIR), 2024
https://scholar.google.com/scholar?q=Better+%26+Faster+Large+Language+Models+via+Multi-token+Prediction
7. EAGLE-3: Scaling up Inference Acceleration of Large…
