Weight Decay Improves Language Model Plasticity

In this episode:
• Introduction and The Premise: Professor Norris and Linda introduce the episode's focus: a new paper showing how weight decay impacts language model plasticity.
• Challenging the Validation Loss Paradigm: Linda explains how optimizing for pretraining validation loss can actually hurt downstream fine-tuning performance.
• The Mechanics of Plasticity: A deep dive into why weight decay helps, covering linearly separable representations, attention matrix rank reduction, and the prevention of overfitting.
• Trade-offs and Practical Implications: The hosts discuss the…