Show notes
When an AI model answers slowly, the problem usually isn't its intelligence — it's how the serving system moves data through a GPU. This episode goes one level below the basic inference loop covered in an earlier episode, into GPU memory, the key-value cache, attention kernels, and the scheduling techniques used by systems like vLLM and NVIDIA TensorRT-LLM.
It starts by separating prefill, which processes prompt tokens in parallel and benefits from raw compute throughput, from decode, which is autoregressive and becomes memory-bound because each step needs the model weights and the growing attention state for every prior token. The episode works through a concrete KV-cache size estimate: with 32 transformer layers, 8 key-value heads, a head dimension of 128, and 2 bytes per cached value, each token costs 131,072 bytes, or 128 kibibytes, of cache; at 8,192 tokens, one sequence's cache runs to roughly a gibibyte before overhead, showing why long context and concurrent users turn KV-cache capacity into a scheduling problem.
From there, it covers the main optimizations serving engines use: grouped-query attention (GQA), which shares key-value heads across multiple query heads to shrink the cache; FlashAttention, which computes exact attention while cutting memory traffic through tiling and on-chip memory; continuous batching, which replaces finished requests in an active batch instead of waiting for a fixed batch to clear; chunked prefill, which splits long prompts into pieces interleaved with decode work so one long prompt doesn't stall other users; PagedAttention, which allocates KV-cache memory in non-contiguous blocks to reduce fragmentation; prefix caching, which reuses cached state across requests sharing an identical prompt prefix; and quantization, which trades numerical precision for memory and speed.
The episode closes with the metrics that tie these optimizations to specific symptoms: time to first token for queueing and prefill issues, inter-token latency and throughput for decode and batching issues, and memory use and tail latency under concurrency for capacity issues, arguing that LLM inference is as much a hardware-and-scheduling problem as a model problem.
Sources & References
NVIDIA Technical Blog: Co-Designing AI Model Attention for Fast, Interactive Long-Context Inference — https://developer.nvidia.com/blog/co-designing-ai-model-attention-for-fast-interactive-long-context-inference
NVIDIA Technical Blog: Mastering LLM Techniques: Inference Optimization — https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization
vLLM Documentation: Automatic Prefix Caching — https://docs.vllm.cc/en/latest/design/automatic_prefix_caching.html
vLLM Documentation — https://docs.vllm.cc/en/latest/
NVIDIA TensorRT-LLM Documentation: Paged Attention, IFB, and Request Scheduling — https://nvidia.github.io/TensorRT-LLM/1.2.0rc3/features/paged-attention-ifb-scheduler.html
NVIDIA Technical Blog: Streamlining AI Inference…
Website: podcasters.spotify.com
