Gå til indhold
Global sound
AI Post Transformers

AI Post Transformers

MiniCPM: Small Language Models and Scalable Training Strategies

Episodebeskrivelse

This episode examines MiniCPM, a pair of small language models (1.2B and 2.4B non-embedding parameters) from Tsinghua and Modelbest. The 2.4B model is reported to compete with 7B-class models. It starts with scaling-law background: the loss model L(N, D), Kaplan's and Chinchilla's compute-optimal results, and why over-training a small model can lower inference cost. The central tension is the paper's claim that the optimal ratio is about 192 tokens per parameter, roughly ten times Chinchilla's 20, which the discussion treats as much shakier than the models themselves because it comes from a single fit. The conversation then covers the Warmup-Stable-Decay learning-rate schedule, which keeps a constant rate for most of training and decays only over the last ~10% of tokens. Unlike cosine decay, this lets any stable checkpoint branch into a finished model or resume for continual pre-training. It also covers "model wind tunnel" experiments on tiny models and the muP parameterization that lets hyperparameters tuned at small width transfer to larger models.

Sources:
1. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies — Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, Maosong Sun, 2024
http://arxiv.org/abs/2404.06395
2. Textbooks Are All You Need (Phi-1) — Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Sébastien Bubeck, Yuanzhi Li, et al. (Microsoft Research), 2023
https://scholar.google.com/scholar?q=Textbooks+Are+All+You+Need+%28Phi-1%29
3. TinyLlama: An Open-Source Small Language Model — Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, Wei Lu, 2024
https://scholar.google.com/scholar?q=TinyLlama%3A+An+Open-Source+Small+Language+Model
4. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases — Zechun Liu, Changsheng Zhao, Forrest Iandola, et al. (Meta), 2024
https://scholar.google.com/scholar?q=MobileLLM%3A+Optimizing+Sub-billion+Parameter+Language+Models+for+On-Device+Use+Cases
5. Small Language Models: Survey, Measurements, and Insights — Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, et al., 2024
https://scholar.google.com/scholar?q=Small+Language+Models%3A+Survey%2C+Measurements%2C+and+Insights
6. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective — Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, Tengyu Ma, 2024
https://scholar.google.com/scholar?q=Understanding+Warmup-Stable-Decay+Learning+Rates%3A+A+River+Valley+Loss+Landscape+Perspective
7. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations — Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, Martin Jaggi…

Live, seneste time: 14.277 sange på 7.302 stationer i 145 lande

Opdag verden gennem radioen

Flere måder at opdage på (7)

Hørt noget godt? Gem det, eller åbn det direkte i Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
KTFM / Global SportFortsæt med at lytte