Zum Inhalt springen
Global sound
Best AI papers explained

Best AI papers explained

Multi-Turn LLM Conversations under the Least-Recently-Used Policy: Mean-Field Asymptotics

· 22 Min.

Shownotes

This paper introduces a Mean-Field Asymptotic framework designed to estimate the hit ratio in multi-turn large language model (LLM) serving systems. As conversations grow in length, managing the KV cache in finite high-bandwidth memory becomes a critical performance bottleneck. The authors model these dynamics using the least-recently-used (LRU) eviction policy to determine which conversation histories are retained or discarded. By analyzing the system as memory capacity and arrival rates scale toward infinity, they derive a closed-form limit to accurately predict cache reuse. The study further proposes a practical estimator that accounts for partially filled, unhashable memory blocks common in real-world applications. Finally, the researchers validate their theoretical findings through experiments with the Qwen3-8B model, demonstrating that their model reliably predicts system performance under varying workloads.

Website: podcasters.spotify.com

Live, letzte Stunde: 12.228 Songs auf 4.894 Sendern in 129 Ländern

Entdecke die Welt über das Radio

Mehr zum Entdecken (9)

Etwas Schönes gehört? Speicher es oder öffne es direkt in Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
94.1 KODJWeiterhören