Skip to content
Global sound
Best AI papers explained

Best AI papers explained

A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets

· 24 min

Show notes

This paper analyzes policy optimization in reinforcement learning post-training, focusing on how base models, regularization, and clipping affect optimization and sample complexity. They demonstrate that standard methods like REINFORCE can suffer from severe coverage collapse, failing to reach optimality even with high-quality base models. To resolve this, the text introduces Approximate Policy Mirror Descent (APMD), a meta-algorithm that periodically resets the reference policy to maintain response coverage throughout training. Furthermore, the sources examine specific implementations such as PPO-NoClip, PPO-Lite, and squared-loss objectives like KimiPO and Rebel-OnPolicy. Ultimately, the research provides theoretical guarantees showing that applying strong regularization alongside reference resets yields stable, efficient, and coverage-preserving global convergence.

Website: podcasters.spotify.com

Live, last hour: 15,692 songs on 7,639 stations in 138 countries

Discover the world through radio

More ways to discover (9)

Heard something good? Save it, or open it straight away in Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
TouchFMContinue listening