Hoppa till innehållet
Global sound
Best AI papers explained

Best AI papers explained

Generalization Dynamics of Language Model Pretraining

· 23 min

Avsnittsbeskrivning

Large language models often exhibit unexpected behavioral fluctuations during pre-training, repeatedly shifting between memorizing shallow patterns and demonstrating true generalization, a phenomenon termed mode-hopping. This instability occurs independently of standard optimization metrics and persists even after training on trillions of tokens across various model scales. Consequently, tracking these hidden generalization dynamics using targeted evaluation suites enables researchers to identify superior intermediate checkpoints and strategically curate pre-training data to improve reasoning and alignment robustness.

Webbplats: podcasters.spotify.com

Live, senaste timmen: 10 411 låtar på 5 647 stationer i 150 länder

Upptäck världen genom radion

Fler sätt att upptäcka (9)

Hört något bra? Spara det, eller öppna det direkt i Spotify.

NasyeedfmFortsätt lyssna