Naar de inhoud
Global sound
AI Post Transformers

AI Post Transformers

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient

Beschrijving

This episode explores the paper "Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient," which asks whether a mixture-of-experts model can beat a dense model on loss when both train with the same compute and are capped at the same total parameter count. It covers how Switch-style top-1 routing works with experts the same size as the dense feed-forward layer. It also covers the difference between active and total parameters, using a 1.1 billion total, 426 million active, four-expert example. The discussion explains how a smaller active size lets an MoE see more tokens for the same FLOPs. That ties into deliberate overtraining under Chinchilla-style tradeoffs, and into a memory definition that includes the KV cache. The hosts also place the work alongside related studies on sparsity, inference-aware scaling and hardware efficiency. They flag the gap between a loss-based claim and real serving hardware as the question to keep in mind.

Sources:
1. Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
https://arxiv.org/pdf/2502.05172
2. Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models — Samira Abnar, Harshay Shah, Dan Busbridge, et al., 2025
https://scholar.google.com/scholar?q=Parameters+vs+FLOPs%3A+Scaling+Laws+for+Optimal+Sparsity+for+Mixture-of-Experts+Language+Models
3. Toward Inference-Optimal Mixture-of-Expert Large Language Models — Longfei Yun, Yonghao Zhuang, Yao Fu, Eric P. Xing, Hao Zhang, 2024
https://scholar.google.com/scholar?q=Toward+Inference-Optimal+Mixture-of-Expert+Large+Language+Models
4. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — Nikhil Sardana, Jacob Portes, Sasha Doubov, Jonathan Frankle, 2024
https://scholar.google.com/scholar?q=Beyond+Chinchilla-Optimal%3A+Accounting+for+Inference+in+Language+Model+Scaling+Laws
5. Unified Scaling Laws for Routed Language Models — Aidan Clark et al., 2022
https://scholar.google.com/scholar?q=Unified+Scaling+Laws+for+Routed+Language+Models
6. Scaling Laws for Fine-Grained Mixture of Experts — Jan Ludziejewski et al., 2024
https://scholar.google.com/scholar?q=Scaling+Laws+for+Fine-Grained+Mixture+of+Experts
7. Reconciling Kaplan and Chinchilla Scaling Laws / Resolving Discrepancies in Compute-Optimal Scaling of Language Models — Tomer Porian et al.; Tim Pearce & Jinyeop Song, 2024-2025
https://scholar.google.com/scholar?q=Reconciling+Kaplan+and+Chinchilla+Scaling+Laws+%2F+Resolving+Discrepancies+in+Compute-Optimal+Scaling+of+Language+Models
8. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations — Alexander Hägele et al., 2024
https://scholar.google.com/scholar?q=Scaling+Laws+and+Compute-Optimal+Training+Beyond+Fixed+Training+Durations
9. OLMoE: Open Mixture-of-Experts Language Models — Niklas Muennighoff et al., 2024
https://scholar.google.com/scholar?q=OLMoE%3A+Open+Mixture-of-Experts+Language+Models
10. Fast Inference of Mixture-of-Experts Language Models…

Live, afgelopen uur: 11.003 nummers op 4.658 zenders in 124 landen

Ontdek de wereld via de radio

Meer manieren om te ontdekken (7)

Iets moois gehoord? Bewaar het, of open het direct in Spotify.

★ Artiest van de week De snelste stijger op de radio deze week
Jouw nummer hier? Vanaf € 9 per week →
Radio Caprice - Gothic MetalVerder luisteren