Shownotes
This episode examines "Scaling Laws for Precision," which argues that numeric format belongs in the planning equation alongside model size and training data, rather than being treated as a deployment detail. It reviews the Chinchilla baseline and why overtraining is now common. It then explains the paper's central idea: lower precision acts like shrinking the model, captured by an "effective parameter count" whose gains flatten around six or seven bits. The discussion separates post-training quantization, quantization-aware training and full low-precision training. It also covers the experimental setup of 465 small-scale pretraining runs and the caveat that the fitted constants are brittle, so the functional forms matter more than the values. The first finding it reaches is that post-training quantization hurts more the longer a model has trained, which challenges the push toward 4-bit and the default of 16-bit training. Listeners get a framework for judging precision-versus-size bets in an era of FP8 and FP4 hardware.
Sources:
1. Scaling Laws for Precision — Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, Aditi Raghunathan, 2024
http://arxiv.org/abs/2411.04330
2. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020
https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models
3. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, and others (DeepMind), 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29
4. FP8 Formats for Deep Learning — Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, and others (NVIDIA, Arm, Intel), 2022
https://scholar.google.com/scholar?q=FP8+Formats+for+Deep+Learning
5. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58) — Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei, 2024
https://scholar.google.com/scholar?q=The+Era+of+1-bit+LLMs%3A+All+Large+Language+Models+are+in+1.58+Bits+%28BitNet+b1.58%29
6. The case for 4-bit precision: k-bit Inference Scaling Laws — Tim Dettmers, Luke Zettlemoyer, 2023
https://scholar.google.com/scholar?q=The+case+for+4-bit+precision%3A+k-bit+Inference+Scaling+Laws
7. Language models scale reliably with over-training and on downstream tasks — Samir Yitzhak Gadre et al., 2024
https://scholar.google.com/scholar?q=Language+models+scale+reliably+with+over-training+and+on+downstream+tasks
8. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws — Nikhil Sardana, Jonathan Frankle…

