Episodebeskrivelse
This episode examines "Soup-of-Experts," an Apple paper on pretraining one bank of parameters that can be averaged into a small specialist model for any data mixture, with no retraining. It sets up the background: clustered importance sampling over 4096 RedPajama-v2 clusters, model soups from fine-tuning, per-domain expert baselines, and the hypernetwork framing. It also notes closely related prior work, Branch-Train-Merge and c-BTM, that the paper doesn't compare against. The discussion then moves to the mechanism: shared parameters plus 128 experts, combined by a small MLP that maps a domain histogram to mixing coefficients. Listeners get a clear look at the amortization argument, which is paying one large pretraining cost so each later specialist is nearly free. They also get a skeptical read on whether the headline comparison holds up.
Sources:
1. Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging — Pierre Ablin, Angelos Katharopoulos, Skyler Seto, David Grangier, 2025
http://arxiv.org/abs/2502.01804
2. Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models — Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Noah A. Smith, Luke Zettlemoyer, 2022
https://scholar.google.com/scholar?q=Branch-Train-Merge%3A+Embarrassingly+Parallel+Training+of+Expert+Language+Models
3. Scaling Expert Language Models with Unsupervised Domain Discovery (c-BTM) — Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, Luke Zettlemoyer, 2023
https://scholar.google.com/scholar?q=Scaling+Expert+Language+Models+with+Unsupervised+Domain+Discovery+%28c-BTM%29
4. Hypernetworks — David Ha, Andrew Dai, Quoc V. Le, 2017
https://scholar.google.com/scholar?q=Hypernetworks
5. HyperTuning / Hypernetwork-based task adaptation (e.g., HyperFormer, Hyper-X) — Various (Mahabadi et al.; Üstün et al.; Phang et al.), 2021-2023
https://scholar.google.com/scholar?q=HyperTuning+%2F+Hypernetwork-based+task+adaptation+%28e.g.%2C+HyperFormer%2C+Hyper-X%29
6. Mixture of Experts Models / Soft MoE (From Sparse to Soft Mixtures of Experts) — Joan Puigcerver, Carlos Riquelme, Basil Mustafa, Neil Houlsby, 2024
https://scholar.google.com/scholar?q=Mixture+of+Experts+Models+%2F+Soft+MoE+%28From+Sparse+to+Soft+Mixtures+of+Experts%29
7. Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training — Zexuan Zhong, Mengzhou Xia, Danqi Chen, Mike Lewis, 2024
https://scholar.google.com/scholar?q=Lory%3A+Fully+Differentiable+Mixture-of-Experts+for+Autoregressive+Language+Model+Pre-training
8. Sheared LLaMA / structured pruning to a target-task model — Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi Chen, 2023
https://scholar.google.com/scholar?q=Sheared+LLaMA+%2F+structured+pruning+to+a+target-task+model
9. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pre-training — Sang Michael Xie et al…

