Notas do episódio
This episode examines Cascade, a serving system that uses a single "latency budget" (a request's time-to-first-token target minus its predicted service time) to drive both queue ordering and KV cache restoration decisions. It explains why first-come-first-served, shortest-job-first and earliest-deadline-first each fail. The failures are head-of-line blocking, starved long prompts, and an inability to tell a 50 ms job from a 20 s job with the same deadline. The discussion covers goodput, tiered prefix caching across GPU memory, CPU DRAM and NVMe, and chunked prefill. It also covers the paper's motivating findings: aggregate SLO attainment hides starved classes, deep-tier restores such as a 64,000-token NVMe fetch can blow the latency target, and most violations come from queueing rather than a lack of budget. Listeners get a clear walkthrough of the scheduling vocabulary, Jain's fairness index and the budget estimator. They also get a caveat that the headline cluster results come from an extended simulator fed with GB200 NVL72 data-movement profiles rather than a real deployment.
Sources:
1. Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving — Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse, 2026
http://arxiv.org/abs/2608.06557
2. Orca: A Distributed Serving System for Transformer-Based Generative Models — Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, Byung-Gon Chun, 2022
https://scholar.google.com/scholar?q=Orca%3A+A+Distributed+Serving+System+for+Transformer-Based+Generative+Models
3. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024
https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve
4. Fast Distributed Inference Serving for Large Language Models (FastServe) — Bingyang Wu, Yinmin Zhong, Zili Zhang, Shengyu Liu, Fangyue Liu, Yuanhang Sun, Gang Huang, Xuanzhe Liu, Xin Jin, 2023
https://scholar.google.com/scholar?q=Fast+Distributed+Inference+Serving+for+Large+Language+Models+%28FastServe%29
5. Efficient LLM Scheduling by Learning to Rank — Yichao Fu, Siqi Zhu, Runlong Su, Aurick Qiao, Ion Stoica, Hao Zhang, 2024
https://scholar.google.com/scholar?q=Efficient+LLM+Scheduling+by+Learning+to+Rank
6. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
7. Fairness in Serving Large Language Models — Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica…
