Note de episod
This episode examines "Learning What to Remember: Test-Time Training via Context Distillation" (TTCD), which asks what a model should write into a fixed-size memory when it can't keep everything. It starts with why linear-attention and recurrent models like DeltaNet and Gated DeltaNet need lossy bounded state. It then traces the progression of test-time training from the delta rule through TTT layers, Titans, LaCT, and In-Place TTT, which reuses a transformer's MLP down-projection as fast weights. The paper's method runs the same frozen backbone twice, once with a long sliding window as teacher and once with a short window as student. It writes the residual between their MLP outputs into the fast weights through an outer-product update, so the information outlives the short window. The discussion stays skeptical. It asks whether the residual really captures remote history or just a modest window extension, whether the distillation loss actually shapes the writes, and why the closest next-token competitor, TTT-E2E, is left out of the comparisons.
Sources:
1. Learning What to Remember: Test-Time Training via Context Distillation — Zixuan Wang, Xingyu Dang, Rui-Jie Zhu, Zixin Wen, Hengyu Fu, Wenhao Chai, Jason D. Lee, 2026
http://arxiv.org/abs/2608.01672
2. Learning to (Learn at Test Time): RNNs with Expressive Hidden States — Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin, 2024 (ICML 2025 version also circulated)
https://scholar.google.com/scholar?q=Learning+to+%28Learn+at+Test+Time%29%3A+RNNs+with+Expressive+Hidden+States
3. Gated Delta Networks: Improving Mamba2 with Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh, 2024 (ICLR 2025)
https://scholar.google.com/scholar?q=Gated+Delta+Networks%3A+Improving+Mamba2+with+Delta+Rule
4. Titans: Learning to Memorize at Test Time — Ali Behrouz, Peilin Zhong, Vahab Mirrokni, 2024
https://scholar.google.com/scholar?q=Titans%3A+Learning+to+Memorize+at+Test+Time
5. A General Method for Test-Time Training / Learning by Distilling Context — Snell, Klein, Zhong (Learning by Distilling Context); Askell et al. (A General Language Assistant as a Laboratory for Alignment), 2022 (Snell et al.); 2021 (Askell et al.)
https://scholar.google.com/scholar?q=A+General+Method+for+Test-Time+Training+%2F+Learning+by+Distilling+Context
6. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning — Ekin Akyürek, Mehul Damani, Linlu Qiu, Han Guo, Yoon Kim, Jacob Andreas, 2024
https://scholar.google.com/scholar?q=The+Surprising+Effectiveness+of+Test-Time+Training+for+Few-Shot+Learning
7. End-to-End Test-Time Training for Long Context (TTT-E2E) — Arnuv Tandon, Karan Dalal, Xinhao Li, et al., 2025
https://scholar.google.com/scholar?q=End-to-End+Test-Time+Training+for+Long+Context+%28TTT-E2E%29
8. In-Place Test-Time Training — Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Wenhao Huang, Di He, Tianle…
