Zum Inhalt springen
Global sound
AI Post Transformers

AI Post Transformers

Scaling Laws for Precision: Bits, Parameters, and Training Tokens

Shownotes

This episode examines "Scaling Laws for Precision," which argues that numeric format belongs in the planning equation alongside model size and training data, rather than being treated as a deployment detail. It reviews the Chinchilla baseline and why overtraining is now common. It then explains the paper's central idea: lower precision acts like shrinking the model, captured by an "effective parameter count" whose gains flatten around six or seven bits. The discussion separates post-training quantization, quantization-aware training and full low-precision training. It also covers the experimental setup of 465 small-scale pretraining runs and the caveat that the fitted constants are brittle, so the functional forms matter more than the values. The first finding it reaches is that post-training quantization hurts more the longer a model has trained, which challenges the push toward 4-bit and the default of 16-bit training. Listeners get a framework for judging precision-versus-size bets in an era of FP8 and FP4 hardware.

Sources:
1. Scaling Laws for Precision — Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, Aditi Raghunathan, 2024
http://arxiv.org/abs/2411.04330
2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020
https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models
3. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, and others (DeepMind), 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29
4. FP8 Formats for Deep Learning — Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, and others (NVIDIA, Arm, Intel), 2022
https://scholar.google.com/scholar?q=FP8+Formats+for+Deep+Learning
5. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58) — Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei, 2024
https://scholar.google.com/scholar?q=The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits+%28BitNet+b1.58%29
6. The case for 4-bit precision: k-bit Inference Scaling Laws — Tim Dettmers, Luke Zettlemoyer, 2023
https://scholar.google.com/scholar?q=The+case+for+4-bit+precision%3A+k-bit+Inference+Scaling+Laws
7. Language models scale reliably with over-training and on downstream tasks — Samir Yitzhak Gadre et al., 2024
https://scholar.google.com/scholar?q=Language+models+scale+reliably+with+over-training+and+on+downstream+tasks
8. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — Nikhil Sardana, Jonathan Frankle…

Live, letzte Stunde: 6.757 Songs auf 2.891 Sendern in 126 Ländern

Entdecke die Welt über das Radio

Mehr zum Entdecken (7)

Etwas Schönes gehört? Speicher es oder öffne es direkt in Spotify.

★ Künstler der Woche Der schnellste Aufsteiger im Radio diese Woche
Dein Song hier? Ab € 9 pro Woche →
Radio 538 luisteren - Live!Gerade beliebt