Episodebeskrivelse
This episode examines "Cartridges++," an Apple paper on learned, compressed KV-cache memories of a document (Cartridges) and a failure they cause on questions unrelated to that document. It explains how KV-cache compression works and compares three families: eviction methods like H2O and SnapKV, text summaries, and learned methods such as Cartridges, prefix-tuning and Attention Matching. The central argument is that a Cartridge's training objective only rewards matching the teacher on document-related queries and says nothing about unrelated ones. So the memory that best reproduces the document can derail the model elsewhere, for example by quoting a stored policy document when asked what Saturn is made of. The discussion separates context fidelity from capability preservation and flags interference, meaning fabricated quotes from the document. It argues the fair baseline is the full uncompressed document pasted as text, which already distracts models somewhat. Listeners get a clear account of why scoring a compressed memory on document questions alone hides a real cost, and why the problem looks like catastrophic forgetting applied to cache entries.
Sources:
1. Cartridges++: KV Cache Compression without Off-Context Derailment — Sonia Laguna, Joao Monteiro, Marco Cuturi, Pierre Ablin, Eleonora Gualdoni, 2026
http://arxiv.org/abs/2609.35621
2. Cartridges: Lightweight and General-Purpose Long Context Representations via Self-Study — Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, Christopher Ré, 2025
https://scholar.google.com/scholar?q=Cartridges%3A+Lightweight+and+General-Purpose+Long+Context+Representations+via+Self-Study
3. Overcoming catastrophic forgetting in neural networks — James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell, 2017
https://scholar.google.com/scholar?q=Overcoming+catastrophic+forgetting+in+neural+networks
4. LoRA Learns Less and Forgets Less — Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, and colleagues (Databricks / Columbia), 2024
https://scholar.google.com/scholar?q=LoRA+Learns+Less+and+Forgets+Less
5. Large Language Models Can Be Easily Distracted by Irrelevant Context — Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, Denny Zhou, 2023
https://scholar.google.com/scholar?q=Large+Language+Models+Can+Be+Easily+Distracted+by+Irrelevant+Context
6. Fast KV Compaction via Attention Matching — Adam Zweiger et al., 2026
https://scholar.google.com/scholar?q=Fast+KV+Compaction+via+Attention+Matching
7. Cartridges at Scale — Hardalov et al., 2026
https://scholar.google.com/scholar?q=Cartridges+at+Scale
8. Prefix-Tuning: Optimizing Continuous Prompts for Generation — Xiang Lisa Li, Percy Liang…
