Episodebeskrivelse
This episode explores Context Language Models, a proposal that lets a model write directly to its own context as an editable file instead of relying on human-designed harnesses. It first surveys the existing approaches: fixed-policy compaction, action-based tools such as Context Folding, and recursive language models. It then shows why they fall short on tasks like in-place Sudoku edits, using a diagnostic benchmark that varies context pressure. The discussion explains why edits to the middle of a context break prefix caching and force costly recomputation, and why the paper measures cost in prefix-reuse FLOPs. It also lays out the paper's five claims: zero-shot gains on deep research and coding, swarms of agents with file-based contexts, evolved natural-language skills, online reinforcement learning, and a serving change called Suffix Cache Reuse. The hosts stress the difference between percentage points and relative percent, and they note that the strongest evidence is zero-shot with a large model while the training evidence is thinner.
Sources:
1. Context Language Models — Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Pang Wei Koh, 2026
http://arxiv.org/abs/2609.37725
2. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO) — Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. (DeepSeek-AI), 2024
https://scholar.google.com/scholar?q=DeepSeekMath%3A+Pushing+the+Limits+of+Mathematical+Reasoning+in+Open+Language+Models+%28introduces+GRPO%29
3. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents — Zijian Zhou et al., 2025
https://scholar.google.com/scholar?q=MEM1%3A+Learning+to+Synergize+Memory+and+Reasoning+for+Efficient+Long-Horizon+Agents
4. Scaling Long-Horizon LLM Agent via Context-Folding — Weiwei Sun et al. (ByteDance Seed / CMU), 2025
https://scholar.google.com/scholar?q=Scaling+Long-Horizon+LLM+Agent+via+Context-Folding
5. BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent — Zijian Chen, Xueguang Ma, Shengyao Zhuang, et al. (Waterloo and collaborators), 2025
https://scholar.google.com/scholar?q=BrowseComp-Plus%3A+A+More+Fair+and+Transparent+Evaluation+Benchmark+of+Deep-Research+Agent
6. MemGPT: Towards LLMs as Operating Systems — Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, Joseph E. Gonzalez, 2023
https://scholar.google.com/scholar?q=MemGPT%3A+Towards+LLMs+as+Operating+Systems
7. How we built our multi-agent research system (Anthropic engineering report) / Effective context engineering for AI agents — Anthropic engineering team, 2025
https://scholar.google.com/scholar?q=How+we+built+our+multi-agent+research+system+%28Anthropic+engineering+report%29+%2F+Effective+context+engineering+for+AI+agents
8. Don't Build Multi-Agents (Cognition AI blog post) — Walden…
