Poznámky k epizodě
In this episode:
• Introduction to Higher-Order Optimizers: Linda introduces the NVIDIA paper on SOAP and Muon, and Professor Norris expresses his initial skepticism about dethroning AdamW.
• The Batch Size Wall and MoEs: The hosts discuss why AdamW struggles at massive batch sizes, particularly for the dense layers in Mixture of Experts models.
• Muon and Spectral Descent: Linda explains how Muon orthogonalizes momentum using Newton-Schulz iterations to maintain stability at scale.
• Fixing SOAP's Slingshot Instability: The conversation shifts to SOAP, diagnosing the stale preconditioner…
Web: mechanicaldreams.mechanicaldirk.com

