Notas del episodio
This episode examines "Averaging Weights Leads to Wider Optima and Better Generalization," which proposes Stochastic Weight Averaging (SWA): instead of decaying the learning rate to near zero, keep it high or cyclical and take an equal-weight average of the weights SGD visits. The discussion builds the background step by step. It covers mode connectivity, which suggests good solutions form a connected region rather than isolated pits. It covers Fast Geometric Ensembling, whose snapshots sit on the periphery of that region. And it covers the link between wide minima and generalization, including caveats from Entropy-SGD and from the rescaling argument for ReLU/BatchNorm networks. The episode also contrasts SWA with exponential moving averages and classical Polyak–Juditsky averaging. It walks through the practical recipe: train normally for about three quarters of the budget, continue at a high or cyclical rate, fold weights into a running average once per epoch, then recompute BatchNorm statistics. Listeners get a skeptical framing around three claims (accuracy gains, the width story, and approximating an ensemble at single-model serving cost) along with the question of whether the baseline is fair.
Sources:
1. Averaging Weights Leads to Wider Optima and Better Generalization — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018
http://arxiv.org/abs/1803.05407
2. Averaging Weights Leads to Wider Optima and Better Generalization — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (UAI)
https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization
3. Acceleration of Stochastic Approximation by Averaging — Boris Polyak, Anatoli Juditsky, 1992
https://scholar.google.com/scholar?q=Acceleration+of+Stochastic+Approximation+by+Averaging
4. Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy without Increasing Inference Time — Mitchell Wortsman, Gabriel Ilharco, et al., 2022
https://scholar.google.com/scholar?q=Model+Soups%3A+Averaging+Weights+of+Multiple+Fine-Tuned+Models+Improves+Accuracy+without+Increasing+Inference+Time
5. Fast Geometric Ensembling / Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs — Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (NeurIPS)
https://scholar.google.com/scholar?q=Fast+Geometric+Ensembling+%2F+Loss+Surfaces%2C+Mode+Connectivity%2C+and+Fast+Ensembling+of+DNNs
6. Flat Minima — Sepp Hochreiter, Jürgen Schmidhuber, 1997
https://scholar.google.com/scholar?q=Flat+Minima
7. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang, 2017 (ICLR)
https://scholar.google.com/scholar?q=On+Large-Batch+Training+for+Deep+Learning%3A+Generalization+Gap+and+Sharp+Minima
8. Sharp Minima Can Generalize For Deep…

