Catatan episode
This episode examines "Vector Policy Optimization," a proposed drop-in replacement for the GRPO advantage that trains models to produce diverse candidates rather than a single best answer, so that test-time search such as best-of-N and evolutionary code search has more to work with. It lays out why standard RL post-training collapses onto one high-reward mode. The evidence cited is that RL models beat base models at small k but fall behind at large k, and that KL-regularized RL is argued to collapse structurally. The method treats rewards as vectors, such as per-test-case or per-hop scores. It samples random scalarization weights from a uniform Dirichlet during training, so candidates can land at different points on the Pareto frontier. The discussion compares this with pass@k optimization, entropy bonuses and embedding-based diversity rewards, and with a multi-answer baseline that gives the model room to diversify but no incentive to do so. It also defines the evaluation metrics, pass@k, best@k and reward-space diversity. For listeners, the interest is the argument that giving up some single-shot accuracy can pay off once a search procedure sits on top of the model.
Sources:
1. Vector Policy Optimization: Training for Diversity Improves Test-Time Search — Ryan Bahlous-Boldi, Isha Puri, Idan Shenfeld, Akarsh Kumar, Mehul Damani, Sebastian Risi, Omar Khattab, Zhang-Wei Hong, Pulkit Agrawal, 2026
http://arxiv.org/abs/2605.22817
2. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — Zhihong Shao, Peiyi Wang, Qihao Zhu, et al. (DeepSeek-AI), 2024
https://scholar.google.com/scholar?q=DeepSeekMath%3A+Pushing+the+Limits+of+Mathematical+Reasoning+in+Open+Language+Models
3. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? — Yang Yue, Zhiqi Chen, Rui Lu, et al., 2025
https://scholar.google.com/scholar?q=Does+Reinforcement+Learning+Really+Incentivize+Reasoning+Capacity+in+LLMs+Beyond+the+Base+Model%3F
4. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling — Bradley Brown, Jordan Juravsky, Ryan Ehrlich, et al., 2024
https://scholar.google.com/scholar?q=Large+Language+Monkeys%3A+Scaling+Inference+Compute+with+Repeated+Sampling
5. Understanding the Effects of RLHF on LLM Generalisation and Diversity — Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, et al., 2024
https://scholar.google.com/scholar?q=Understanding+the+Effects+of+RLHF+on+LLM+Generalisation+and+Diversity
6. A Survey of Multi-Objective Sequential Decision-Making — Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, Richard Dazeley, 2013
https://scholar.google.com/scholar?q=A+Survey+of+Multi-Objective+Sequential+Decision-Making
7. A Practical Guide to Multi-Objective Reinforcement Learning and Planning — Conor F. Hayes, Roxana Rădulescu, Eugenio Bargiacchi, et al., 2022
https://scholar.google.com/scholar?q=A+Practical+Guide+to+Multi-Objective+Reinforcement+Learning+and+Planning
8. A…