Show notes
This episode examines FlatCake, a proposed KV cache storage architecture for LLM serving that drops the usual GPU, host DRAM, and NVMe SSD hierarchy. Instead of migrating blocks between tiers, it decides at write time whether each block lives in DRAM or on SSD, and it never moves the block afterward. The discussion explains why the authors think tiering breaks down. Reuse is bimodal: in several conversational and API traces, most blocks are used only once. In a Claude Code agentic trace, sessions are so large that a few concurrent ones overflow DRAM. Staging buffers also run out under load, and blocks written in eviction order end up scattered across the drive. The hosts walk through the proposed fix, which lays each session out contiguously on the SSD so a restore becomes a few large reads, and they note that GPUDirect Storage fixes the bounce buffer but not the scatter. They also point out the cost of the approach: placement relies on hand-tuned thresholds with no published values, and a wrong placement choice sticks. Listeners interested in serving infrastructure and storage layout get a critical look at how much the paper's experiments actually test its claims.
Sources:
1. FlatCake: A Flat KVCache Storage Architecture for LLM Serving — Xin Huang, Teng Ma, Zhuoran Li, Yibo Zhao, Zhaoshi Li, Qianchao Zhu, Wei Yang, Chenxi Wang, Tao Xie, Youwei Zhuo, 2026
http://arxiv.org/abs/2610.38355
2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu (Moonshot AI / Tsinghua), 2024 (FAST 2025)
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
3. Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention (AttentionStore) — Bin Gao, Zhuomin He, Puru Sharma, Qingxuan Kang, Djordje Jevdjic, Junbo Deng, Xingkun Yang, Zhou Yu, Pengfei Zuo, 2024 (USENIX ATC 2024)
https://scholar.google.com/scholar?q=Cost-Efficient+Large+Language+Model+Serving+for+Multi-turn+Conversations+with+CachedAttention+%28AttentionStore%29
4. LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference — Yihua Cheng, Yuhan Liu, Jiayi Yao, Yuwei An, Xiaokun Chen, Shaoting Feng, Yuyang Huang, Samuel Shen, Kuntai Du, Junchen Jiang et al., 2025
https://scholar.google.com/scholar?q=LMCache%3A+An+Efficient+KV+Cache+Layer+for+Enterprise-Scale+LLM+Inference
5. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU — Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang, 2023 (ICML 2023)
https://scholar.google.com/scholar?q=FlexGen%3A+High-Throughput+Generative+Inference+of+Large+Language+Models+with+a+Single+GPU
6. ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning — Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He, 2021 (SC…