Saltar al contenido
Global sound
AI Post Transformers

AI Post Transformers

DepthBench: How Residual Connections Unlock Computational Depth in Transformers

Notas del episodio

This episode examines DepthBench, a study of why stacking more transformer layers often adds little useful computation, and which residual and normalization designs fix that when parameter count and training recipe are held constant. It reviews the Pre-LN "Curse of Depth" argument, in which growing residual-stream magnitude makes late layers act like identity. It then surveys ten architectures in three families: normalization tweaks such as LayerNorm Scaling and DeepNorm, multi-stream residuals such as Hyper-Connections and mHC, and cross-layer access such as Attention Residuals. The central claim is that the benefit of going deeper and narrower depends on the architecture: it can hurt under standard Pre-LN but keeps helping under two other designs. That would mean the common shallow-wide shapes are partly an artifact of the residual design. The discussion sets this against Kaplan-style scaling results, layer-pruning studies, and the rival explanation that deep models are simply mis-scaled in training. Listeners get a clear framework for "effective computational depth" and for judging whether the evidence supports the claim.

Sources:
1. DepthBench: Measuring How Residual Connections Enable More Computational Depth — Keyu Wang, Yangyi Huang, Jiale Kang, David González-Martínez, Weiyang Liu, Shiwei Liu, 2026
http://arxiv.org/abs/2609.32534
2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al., 2020
https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models
3. Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers — Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler, 2021
https://scholar.google.com/scholar?q=Scale+Efficiently%3A+Insights+from+Pre-training+and+Fine-tuning+Transformers
4. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases — Zechun Liu, Changsheng Zhao, Forrest Iandola, et al. (Meta), 2024
https://scholar.google.com/scholar?q=MobileLLM%3A+Optimizing+Sub-billion+Parameter+Language+Models+for+On-Device+Use+Cases
5. The Curse of Depth in Large Language Models — Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, Shiwei Liu, 2025
https://scholar.google.com/scholar?q=The+Curse+of+Depth+in+Large+Language+Models
6. Hyper-Connections (and follow-up: mHC, Manifold-Constrained Hyper-Connections) — Defa Zhu, Hongzhi Huang, Zihao Huang, et al. (ByteDance Seed) for HC; DeepSeek-AI researchers (Xie et al.) for mHC, 2024-2025
https://scholar.google.com/scholar?q=Hyper-Connections+%28and+follow-up%3A+mHC%2C+Manifold-Constrained+Hyper-Connections%29
7. Do Language Models Use Their Depth Efficiently? — Róbert Csordás, Christopher D. Manning, Christopher Potts, 2025
https://scholar.google.com/scholar?q=Do+Language+Models+Use+Their+Depth+Efficiently%3F
8. The Unreasonable Ineffectiveness of the Deeper Layers — Andrey…

En directo, última hora: 11.567 canciones en 6.294 emisoras de 142 países

Descubre el mundo a través de la radio

Más formas de descubrir (9)

¿Has oído algo bueno? Guárdalo o ábrelo directamente en Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Pause RadioSeguir escuchando