Notas do episódio
This episode examines how to loop a transformer's layers, meaning reuse the same weights several times per token, depending on whether model size, training compute, or decoding speed is the scarce resource. It compares Full-Stack, Middle-Stack, and Per-Layer looping. Full-Stack gives the best quality at fixed parameters, Middle-Stack gives the best return at fixed training FLOPs, and gains flatten or turn unstable past about 16 loops. The 32-loop evidence comes only from the 70M and 100M models. Small per-loop low-rank adapters add only 1 to 4 percent more parameters and help more as loop counts grow, but at comparable FLOPs, doubling true depth still reaches lower loss. The discussion then turns to the decoding-latency problem, where each loop adds a sequential pass. It covers the Parallel Loop Transformer, which uses the previous token's state to run loops in parallel, and why that shift compounds prediction errors and causes magnitude mismatches. It also covers a proposed fix that re-injects the current token's embedding through normalization and a gate. Listeners get a practical guide to matching a looping design to a real resource constraint, with caveats about how far the evidence scales.
Sources:
1. How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference — Chanakya Ekbote, Artidoro Pagnoni, Devendra Singh Sachan, Shruti Bhosale, Mike Lewis, Maryam Fazel-Zarandi, Shashank Rajput, Sujan Kumar Gonugondla, 2026
http://arxiv.org/abs/2610.40325v1
2. Universal Transformers — Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser, 2018
https://scholar.google.com/scholar?q=Universal+Transformers
3. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach — Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, et al., 2025
https://scholar.google.com/scholar?q=Scaling+up+Test-Time+Compute+with+Latent+Reasoning%3A+A+Recurrent+Depth+Approach
4. Reasoning with Latent Thoughts: On the Power of Looped Transformers — Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, Sashank Reddi, 2025
https://scholar.google.com/scholar?q=Reasoning+with+Latent+Thoughts%3A+On+the+Power+of+Looped+Transformers
5. Parallel Loop Transformer (PLT) — Wu et al. (full title and author list not verified; cited in the paper as Wu et al., 2025), 2025
https://scholar.google.com/scholar?q=Parallel+Loop+Transformer+%28PLT%29
6. LoRA: Low-Rank Adaptation of Large Language Models — Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, 2021
https://scholar.google.com/scholar?q=LoRA%3A+Low-Rank+Adaptation+of+Large+Language+Models
7. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA — Sangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Seungyeon Kim, Tal Schuster, 2024/2025 (cited as Bae et al., 2025 in the paper; verify the exact venue…
