Skip to content
Global sound
AI Post Transformers

AI Post Transformers

Scaling Laws for Fine-Grained Mixture of Experts

Show notes

This episode examines a 2024 paper arguing that mixture-of-experts models outperform dense Transformers at every compute budget, with a gap that widens at scale. That claim contradicts earlier Meta and DeepMind studies, which saw the MoE advantage shrink, and the authors blame two hidden assumptions: a fixed training duration and a fixed expert size. The discussion builds the background from dense feed-forward layers, routers, and the difference between active and total parameters, through the Kaplan and Chinchilla scaling laws. It then introduces "granularity", the ratio of dense feed-forward width to expert width, as a third scaling axis. Granularity is a design choice that was inherited by convention and never shown to be optimal. The episode also stays skeptical of the evidence. The law is fit on a ragged grid of small models (3M to 85M active parameters), yet its headline claims extrapolate to a trillion active parameters, so listeners get a clear view of both the idea and its limits.

Sources:
1. Scaling Laws for Fine-Grained Mixture of Experts — Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, Sebastian Jaszczur, 2024
http://arxiv.org/abs/2402.07871
2. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models — Damai Dai, Chengqi Deng, Chenggang Zhao, et al. (DeepSeek-AI), 2024
https://scholar.google.com/scholar?q=DeepSeekMoE%3A+Towards+Ultimate+Expert+Specialization+in+Mixture-of-Experts+Language+Models
3. Scaling Laws for Fine-Grained Mixture of Experts — Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Sebastian Jaszczur, et al., 2024
https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts
4. Unified Scaling Laws for Routed Language Models — Aidan Clark, Diego de las Casas, Aurelia Guy, et al. (DeepMind), 2022
https://scholar.google.com/scholar?q=Unified+Scaling+Laws+for+Routed+Language+Models
5. Mixture of A Million Experts — Xu Owen He (Google DeepMind), 2024
https://scholar.google.com/scholar?q=Mixture+of+A+Million+Experts
6. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, et al. (OpenAI), 2020
https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models
7. Training Compute-Optimal Large Language Models — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. (DeepMind), 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models
8. Scaling Laws for Sparsely-Connected Foundation Models / Scaling Laws for MoE variants (e.g., Artetxe et al., 'Efficient Large Scale Language Modeling with Mixtures of Experts') — Mikel Artetxe, Shruti Bhosale, Naman Goyal, et al. (Meta AI)…

Live, last hour: 8,732 songs on 3,508 stations in 130 countries

Discover the world through radio

More ways to discover (7)

Heard something good? Save it, or open it straight away in Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
FRECUENCIA CCIContinue listening