Note de episod
This research paper introduces a "Sleep" paradigm for Large Language Models (LLMs) to overcome the limitations of static knowledge and catastrophic forgetting. Inspired by human biology, the framework alternates between active phasesfor processing external data and sleep phases for internal knowledge refinement. During sleep, the model performs memory consolidation by expanding its parameters and distilling fragile, short-term information into stable, long-term parametric memory. A secondary "dreaming" phase utilizes reinforcement learning and synthetic data generation to enable recursive self-improvement without human oversight. Technical contributions include knowledge seeding, an upward distillation process, and a generalized distillation objective that combines imitation learning with on-policy data. Experimental results demonstrate that this lifecycle significantly enhances performance in continual learning, factual knowledge incorporation, and long-context understanding.
Site: podcasters.spotify.com

