Gå til indhold
Global sound
AI Post Transformers

AI Post Transformers

Model Soups: Averaging Fine-Tuned Weights Beats Ensembles' Cost

Episodebeskrivelse

This episode examines "Model soups," a paper showing that averaging the weights of several fine-tuned models can raise accuracy without adding any inference cost. It explains why this should be risky: the loss landscape has many basins, and averaging across a barrier breaks the network. Models fine-tuned from the same pre-trained checkpoint tend to share a basin, which is what makes the averaging work. The discussion covers the uniform soup and the greedy soup. The greedy soup adds checkpoints in order of validation accuracy and keeps one only if accuracy doesn't drop. It also compares soups against ensembles, which need k forward passes, and against stochastic weight averaging, which averages along a single training run. Listeners get a clear account of the ImageNet distribution-shift benchmarks, the oracle baseline, and an open question about whether the gains on shifted data come from soups simply pulling weights back toward the initialization.

Sources:
1. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt, 2022
http://arxiv.org/abs/2203.05482
2. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time — Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt, 2022 (ICML)
https://scholar.google.com/scholar?q=Model+soups%3A+averaging+weights+of+multiple+fine-tuned+models+improves+accuracy+without+increasing+inference+time
3. Robust fine-tuning of zero-shot models (WiSE-FT) — Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, Ludwig Schmidt, 2021 (CVPR 2022)
https://scholar.google.com/scholar?q=Robust+fine-tuning+of+zero-shot+models+%28WiSE-FT%29
4. Averaging Weights Leads to Wider Optima and Better Generalization (Stochastic Weight Averaging) — Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, Andrew Gordon Wilson, 2018 (UAI)
https://scholar.google.com/scholar?q=Averaging+Weights+Leads+to+Wider+Optima+and+Better+Generalization+%28Stochastic+Weight+Averaging%29
5. Merging Models with Fisher-Weighted Averaging — Michael Matena, Colin Raffel, 2022 (NeurIPS)
https://scholar.google.com/scholar?q=Merging+Models+with+Fisher-Weighted+Averaging
6. Do Better ImageNet Models Transfer Better? — Simon Kornblith, Jonathon Shlens, Quoc V. Le, 2019 (CVPR)
https://scholar.google.com/scholar?q=Do+Better+ImageNet+Models+Transfer+Better%3F
7. Learning Transferable Visual Models From Natural Language Supervision (CLIP) — Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh…

Live, seneste time: 11.577 sange på 4.554 stationer i 122 lande

Opdag verden gennem radioen

Flere måder at opdage på (9)

Hørt noget godt? Gem det, eller åbn det direkte i Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Dale RadioFortsæt med at lytte