Show notes
This episode examines FlashOffloading, a system that keeps the entire KV cache in host DRAM during both prefill and decode for long-context serving on Ascend 910C chips. It starts with the memory problem: GLM-5.2 needs about 95 KB of KV per token, so a single million-token request needs roughly 97.5 GB. Each decode die has only about 28 GB for KV, so HBM limits concurrency before compute does. The discussion explains why sparse attention with a native top-2048 token indexer makes offloading practical. Each step touches only a small subset of tokens, and attention over them stays exact. The episode then covers the three-part design: rolling layer buffers for prefill, a reverse index with a fused kernel for decode cache management, and a kernel that overlaps miss fetches with attention on hits. It also tests the headline claims of 1.38x prefill and 1.91x decode throughput under a 30 ms per-token budget by asking what the baselines were allowed to do. Listeners get a clear explanation of TTFT, TPOT, ITL, speculative decoding and prefill-decode disaggregation, and of how this work differs from earlier offloading systems such as FlexGen, Quest and Mooncake.
Sources:
1. FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving — Xuan Wang, Zhuanhao Wu, Tingzhu Fang, Xushi Li, Sicheng Zhang, Zhenyu Yang, Feng Liu, Zhaohui Fan, Qiang Xie, Huayi Zou, Yuncong Liu, Zeyu Song, Ke Zhu, Yanjia Zeng, Jingchun Gao, Yan Shi, Kui Luo, Gezi Li, Liangjun Feng, Cheng Li, 2026
http://arxiv.org/abs/2611.07542
2. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, et al., 2023
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
3. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision — Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, Pradeep Ramani, Tri Dao, 2024
https://scholar.google.com/scholar?q=FlashAttention-3%3A+Fast+and+Accurate+Attention+with+Asynchrony+and+Low-precision
4. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention — DeepSeek-AI, 2025
https://scholar.google.com/scholar?q=DeepSeek-V3.2-Exp%3A+Boosting+Long-Context+Efficiency+with+DeepSeek+Sparse+Attention
5. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference — Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han, 2024
https://scholar.google.com/scholar?q=Quest%3A+Query-Aware+Sparsity+for+Efficient+Long-Context+LLM+Inference
6. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al. (Moonshot AI), 2024
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
7. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He…
