Poznámky k epizodě
In this episode:
• Introduction to Distillation and Inference Costs: Linda and Norris introduce the paper and discuss why compute-optimal models are driving the need for smaller, distilled models to reduce inference costs.
• The Distillation Scaling Law and the Capacity Gap: The hosts break down the empirical scaling law discovered in the paper and explore the counter-intuitive capacity gap where overly capable teachers degrade student performance.
• Compute-Optimal Distillation Strategies: Linda explains the economic reality of distillation, detailing when it is actually more…
Web: mechanicaldreams.mechanicaldirk.com

