Σημειώσεις επεισοδίου
In this episode:
• The Data Wall: Introduction to the problem of running out of fresh high-quality pretraining data and the limitations of classical Chinchilla scaling.
• The Compute-Data Scaling Law: Linda introduces the new CD-scaling law, focusing on the token effectiveness function that bridges compute-optimal and data-optimal scaling.
• Eta and the Saturation Ceiling: A deep dive into how data effectiveness decays with model size and data availability, culminating in the R-star saturation ceiling.
• Paraphrasing vs Repetition: Discussion on practical data expansion strategies, revealing…
Ιστότοπος: mechanicaldreams.mechanicaldirk.com
