Why Larger Models Learn More

In this episode:
• Introduction: The Mystery of Scaling: Professor Norris and Linda introduce the core question of why larger models learn tasks that smaller ones cannot, challenging the standard sample-efficiency dogma.
• The Statistical Bottleneck and Task Utility: Linda explains the paper's synthetic regression setup, detailing how models prioritize tasks based on a utility metric derived from task frequency and complexity.
• Gradient Interference and The Update-and-Forget Loop: The hosts dive into the learning dynamics, discussing how frequent tasks overwrite the gradients of rare tasks…