Note de episod
In this episode:
• Introduction to Quantization and Outliers: Norris and Linda introduce the episode's paper, QuaRot, and discuss the main hurdle in LLM inference: the memory bottleneck and the pesky outlier features in activations.
• The Magic of Hadamard Rotations: Linda explains the core mechanism of QuaRot, using randomized Hadamard transformations to eliminate outliers through computational invariance.
• Taming the Attention Mechanism and KV Cache: The hosts dive into the complexities of quantizing the attention module, discussing how QuaRot rotates keys and values to achieve a fully…
Site: mechanicaldreams.mechanicaldirk.com