Episodebeskrivelse
This episode examines "Latent Compaction with Test-Time Training," a preprint that asks whether a frozen model can shrink a document's KV cache by 16 to 32 times by running gradient descent on a small set of new key-value entries. The compaction happens before any questions are known. The discussion explains why the cache bottlenecks long-context memory and batching, and why text summaries lose details like file paths or IDs. It then contrasts the approach with eviction methods such as H2O, SnapKV, PyramidKV and KVZip, which can only select existing entries and never create blended ones. It also places the work among prefix-tuning, Cartridges, Attention Matching and other test-time training approaches, and clarifies that the "32x" figure counts cache slots, not bytes, with no measured speedup or memory savings. Listeners get a clear primer on query-aware versus query-agnostic compression, along with a skeptical look at what the headline RULER result does and doesn't show.
Sources:
1. Latent Compaction with Test-Time Training — Yingfa Chen, Xingyu Shen, Chaojun Xiao, Xu Han, 2026
http://arxiv.org/abs/2610.20582v1
2. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts — Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, Moritz Hardt, 2020
https://scholar.google.com/scholar?q=Test-Time+Training+with+Self-Supervision+for+Generalization+under+Distribution+Shifts
3. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, 2024
https://scholar.google.com/scholar?q=Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States
4. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning — Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, Jacob Andreas, 2024
https://scholar.google.com/scholar?q=The+Surprising+Effectiveness+of+Test-Time+Training+for+Few-Shot+Learning
5. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024
https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time
6. H2O: Heavy-Hitter Oracle: Efficient Generative Inference of Large Language Models with Heavy Hitters — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Zhangyang Wang, Beidi Chen, et al., 2023
https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle%3A+Efficient+Generative+Inference+of+Large+Language+Models+with+Heavy+Hitters
7. SnapKV: LLM Knows What You are Looking for Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, Deming Chen, 2024
https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+are+Looking+for+Before+Generation
8. KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction — Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon…
