İçeriğe geç
Global sound
System Design on the Road

System Design on the Road

S2 E19 — Design an LLM Inference Service: Batching, Admission Control, and Fairness

· 33 dk · Sezon 2 Bölüm 19

Bölüm notları

A short question waits behind a prompt containing an entire book. Daniel and Maya design a shared inference service that uses GPUs efficiently while protecting latency and tenant fairness.

Batch formation, request length, resource budgets, admission, and queueing shape the working path. Slow consumers and worker failures complicate an answer already streaming to a client. The final design explains which throughput gains are worth their waiting time and what can still be promised after partial output.

Chapters
00:00:00 Welcome and a blocked short request
00:02:45 Latency contracts and…

Web sitesi: podcasters.spotify.com

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Heart Kent 103.1 & 102.8Dinlemeye devam et