Continuous-Query Limited Memory Language Models

In this episode:
• Welcome & The Parametric Memory Bottleneck: Professor Norris and Linda introduce the episode's topic, discussing the fundamental flaws of storing facts in model weights.
• From Relational to Continuous Queries: Linda explains the limitations of prior Limited Memory Language Models (LMLMs) and introduces Co-LMLM's continuous-query approach.
• Under the Hood: Training and The Token: The hosts dive into the technical details of how the model uses hidden states as retrieval queries via a special token.
• Scaling Up: The Annotation Pipeline: Discussion on how the researchers…