Show notes
Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on natural languages. It is also unclear whether they can be beaten by training with other objectives. Matthieu Wyart will review the efforts of the collaboration in studying this phenomenon, including the first such theory in the case of data-limited scaling laws. Wyart will then introduce synthetic languages displaying a hidden hierarchy (a central…

