Přejít k obsahu
Global sound
AI Post Transformers

AI Post Transformers

On-Policy Self-Distillation Using Majority Vote Instead of Ground Truth

Poznámky k epizodě

This episode examines "On-Policy Self-Distillation without Any Supervision," a 2026 preprint that asks whether a model can build its own teacher from agreement among its sampled answers, with no ground-truth solutions. It first lays out the ladder of training methods: supervised fine-tuning, RLVR and GRPO, classic on-policy distillation, and on-policy self-distillation, which still needs gold solutions. It also covers majority voting, as in self-consistency and TTRL. The method samples eight answers per prompt and takes a majority vote. It skips prompts where the vote is weak or unanimous, uses the longest agreeing rollout as the teacher's reference, and trains the student only on rollouts that disagreed with the vote. The discussion stresses caveats: the pseudo-labels matched gold only 86.7 percent of the time on a 64-prompt probe, and small benchmarks like AIME are noisy. It also notes that non-thinking Qwen3 has far more headroom than thinking mode. Listeners get a clear explanation of why a noisy self-generated answer key can still help, and of how far that claim should be trusted.

Sources:
1. On-Policy Self-Distillation without Any Supervision — Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian, Di Fu, Nuno Vasconcelos, 2026
http://arxiv.org/abs/2608.06296
2. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD) — Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, Olivier Bachem, 2024 (ICLR)
https://scholar.google.com/scholar?q=On-Policy+Distillation+of+Language+Models%3A+Learning+from+Self-Generated+Mistakes+%28GKD%29
3. Self-Consistency Improves Chain of Thought Reasoning in Language Models — Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Denny Zhou et al., 2022 (ICLR 2023)
https://scholar.google.com/scholar?q=Self-Consistency+Improves+Chain+of+Thought+Reasoning+in+Language+Models
4. TTRL: Test-Time Reinforcement Learning — Yuxin Zuo, Kaiyan Zhang, et al. (Tsinghua / Shanghai AI Lab), 2025
https://scholar.google.com/scholar?q=TTRL%3A+Test-Time+Reinforcement+Learning
5. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (OPSD) — Zhao et al. (cited in the U-OPSD paper as Zhao et al., 2026a), 2026
https://scholar.google.com/scholar?q=Self-Distilled+Reasoner%3A+On-Policy+Self-Distillation+for+Large+Language+Models+%28OPSD%29
6. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO) — Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, et al. (DeepSeek-AI), 2024
https://scholar.google.com/scholar?q=DeepSeekMath%3A+Pushing+the+Limits+of+Mathematical+Reasoning+in+Open+Language+Models+%28introduces+GRPO%29
7. Self-Distillation Enables Continual Learning (SDFT) — Idan Shenfeld et al., 2026
https://scholar.google.com/scholar?q=Self-Distillation+Enables+Continual+Learning+%28SDFT%29
8. Reinforcement Learning via Self-Distillation (SDPO) — Jonas Hübotter, Frederike Lübeck, Lejs Deen Behric…

Živě, poslední hodina: 7 245 skladeb na 2 725 stanicích v 126 zemích

Objevte svět prostřednictvím rádia

Další způsoby objevování (7)

Slyšeli jste něco pěkného? Uložte si to, nebo to otevřete rovnou ve Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Radio SelfiePokračovat v poslechu