Přejít k obsahu
Global sound
Mechanical Dreams

Mechanical Dreams

Pushing LLM Pretraining Scales

· 22 min

Poznámky k epizodě

In this episode:
• Introduction to Higher-Order Optimizers: Linda introduces the NVIDIA paper on SOAP and Muon, and Professor Norris expresses his initial skepticism about dethroning AdamW.
• The Batch Size Wall and MoEs: The hosts discuss why AdamW struggles at massive batch sizes, particularly for the dense layers in Mixture of Experts models.
• Muon and Spectral Descent: Linda explains how Muon orthogonalizes momentum using Newton-Schulz iterations to maintain stability at scale.
• Fixing SOAP's Slingshot Instability: The conversation shifts to SOAP, diagnosing the stale preconditioner…

Web: mechanicaldreams.mechanicaldirk.com

Živě, poslední hodina: 7 941 skladeb na 3 314 stanicích v 128 zemích

Objevte svět prostřednictvím rádia

Další způsoby objevování (9)

Slyšeli jste něco pěkného? Uložte si to, nebo to otevřete rovnou ve Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
SUTON RadioPokračovat v poslechu