Decode-Branch Transformers

In this episode:
• The Inference Bottleneck: Introduction to the podcast and the fundamental asymmetry between compute-bound prefill and memory-bound decoding in LLMs.
• Enter the Decode-Branch: Linda introduces the Decode-Branch Transformer, explaining how it separates prompt processing from continuation prediction without duplicating the KV cache.
• Asymmetric Attention & Shared Weights: A deep dive into the architectural details, including asymmetric shared-KV attention and how the decode branch leverages grouped execution.
• Mixture of Experts and Router Replay: Discussing the…