AI Post Transformers
von mcgrof
Neueste Folge
FastKVComp: Checking Whether High-Throughput KV Compression Pays Off
FastKVComp: Checking Whether High-Throughput KV Compression Pays Off
Über diesen Podcast
AI-generated podcast where hosts Hal Turing and Dr. Ada Shannon discuss the latest research papers and reports in machine learning, AI systems, and optimization. Featuring honest critical analysis, proper citations, and nerdy humor.
Folgen
Die neuesten 50 von 1.044 Folgen
- FastKVComp: Checking Whether High-Throughput KV Compression Pays OffThis episode examines FastKVComp, a system that speeds up KV-cache compression for disaggregated LLM serving, where the cache built during prefill must cross a 100 to 400 gigabit…
- Context Compaction Cuts Costs: "You Don't Need Long ContextThis episode examines a November 2026 NVIDIA paper arguing that long context windows are largely unnecessary, because later tokens cost far more to generate than early ones…
- Nemotron-Labs-Diffusion: One Model, Three Decoding ModesThis episode examines "Nemotron-Labs-Diffusion," an 8B model that runs in three modes from one set of weights: ordinary autoregressive decoding, block-diffusion parallel…
- Cascade: SLO-Aware Latency Budgets for Fair, High-Goodput LLM ServingThis episode examines Cascade, a serving system that uses a single "latency budget" (a request's time-to-first-token target minus its predicted service time) to drive both queue…
- Genuine 1-Bit Post-Training Quantization for LLMs: All for 1-BitThis episode examines "All for 1-Bit," a paper claiming to compress LLM linear weights to exactly one bit each, metadata included, without retraining. It builds on double binary…
- Adaptive Mixed-Precision Computing Speeds Up Nearest Neighbor SearchThis episode examines ANNS-AMP, a June 2026 paper proposing adaptive mixed-precision computing for approximate nearest neighbor search. It starts with the IVF-PQ index pipeline…
- PagedMoE: Cross-Precision Expert Caching for SSD-Resident MoE InferenceThis episode examines PagedMoE, a system for running a huge mixture-of-experts model (Qwen3.5-397B-A17B) on a single PC with expert weights stored on an SSD. Cache misses leave…
- ECHO: Lossless KV Cache Offloading for Sparse Attention LLMsThis episode examines ECHO, a system from Shanghai Jiao Tong University and collaborators (OSDI 2026) that offloads the KV cache to host memory for serving sparse-attention…
- Joules-to-GDP: Economic Value Produced Per Unit of AI EnergyThis episode examines "Joules-to-GDP," a Stanford-led paper that asks how much economic value AI systems produce per joule of energy when doing real professional work. It lays…
- Intelligence per Watt: Measuring Local AI Efficiency Against the CloudThis episode examines "Intelligence per Watt," a Stanford and Together AI paper that asks whether small local models on laptop-class hardware can take real query load off the…
- LeanSparse: Shrinking GPU Memory for Sparse-Attention LLM ServingThis episode examines LeanSparse, a serving system that cuts the GPU memory needed to hold many long-context conversations on models with native sparse attention, such as…
- How to Loop Your Transformer: Scaling, Specialization, Parallel InferenceThis episode examines how to loop a transformer's layers, meaning reuse the same weights several times per token, depending on whether model size, training compute, or decoding…
- Test-Time Training via Context Distillation: Learning What to RememberThis episode examines "Learning What to Remember: Test-Time Training via Context Distillation" (TTCD), which asks what a model should write into a fixed-size memory when it can't…
- Investor's Last Exam: Benchmarking AI Equity Research Without Future LeakageThis episode examines a paper that builds a benchmark for testing whether AI agents can do equity investment research without seeing the future. It centers on a Tesla task whose…
- RaBitQ-SSD: Split Codes and Pipelined I/O for SSD Vector SearchThis episode examines RaBitQ-SSD, a system for approximate nearest neighbor search over vectors stored on SSD, which the paper says can serve ten billion vectors on one drive at…
- FlatCake: Flat KVCache Storage Beats Tiered Memory HierarchiesThis episode examines FlatCake, a proposed KV cache storage architecture for LLM serving that drops the usual GPU, host DRAM, and NVMe SSD hierarchy. Instead of migrating blocks…
- Speculative Decoding Surprisingly Excels on Sparse MoE ModelsThis episode examines why speculative decoding, long thought to give little benefit for sparse mixture-of-experts models at batch size one, actually helps most in that regime for…
- Agent Memory Costs: Characterizing Write-Time Overhead in Long-Horizon WorkloadsThis episode examines a paper that profiles ten agent-memory systems as systems workloads and splits their cost into construction (the write path), retrieval and generation…
- Harness2: Recursive Self-Improving Agent Harnesses Without VerifiersThis episode examines "Harness 2," a paper on recursive agent harnessing. Its premise is that the wrapper around a frozen model can be improved automatically on a stream of…
- Vector Policy Optimization: Training for Diversity to Improve Test-Time SearchThis episode examines "Vector Policy Optimization," a proposed drop-in replacement for the GRPO advantage that trains models to produce diverse candidates rather than a single…
- Sliding-Window Attention Beats Linear Attention Without Any TrainingThis episode examines a paper claiming that a pretrained LLM can be cut down to a sliding window of the last 64 tokens plus four "attention sink" starting tokens, with no…
- DepthBench: How Residual Connections Unlock Computational Depth in TransformersThis episode examines DepthBench, a study of why stacking more transformer layers often adds little useful computation, and which residual and normalization designs fix that when…
- Gradient Noise Scale Predicts Critical Batch Size for Large-Batch TrainingThis episode examines the OpenAI paper on large-batch training, which asks why ImageNet training stalls at batches of tens of thousands of images while Dota 2 agents train on…
- The Illusion of State in State-Space ModelsThis episode examines "The Illusion of State in State-Space Models," which asks whether S4 and Mamba, despite their recurrent form, can track state any better than transformers…
- FastPersist: Making Per-Iteration Deep Learning Checkpointing Nearly FreeThis episode examines FastPersist, a DeepSpeed paper on making deep learning model checkpointing fast enough to save every training iteration. It lays out why checkpoint time has…
- Latent Compaction with Test-Time Training: Learning Compact KV CachesThis episode examines "Latent Compaction with Test-Time Training," a preprint that asks whether a frozen model can shrink a document's KV cache by 16 to 32 times by running…
- LLM Evolutionary Search Finds Cheaper Quantum Arithmetic CircuitsThis episode examines a paper that uses an LLM-driven evolutionary search (OpenEvolve with Claude Mythos 5) to find quantum arithmetic circuits needing fewer Toffoli gates and…
- SanSi: Looped Typed Decision Models for System 1.5 ThinkingThis episode examines SanSi, a small "typed decision" model that returns a probability for each caller-declared option (such as spam or not spam) in a single forward pass, with…
- Cartridges++: Compressed KV Memory Without Off-Context DerailmentThis episode examines "Cartridges++," an Apple paper on learned, compressed KV-cache memories of a document (Cartridges) and a failure they cause on questions unrelated to that…
- Agentic Coding in the Wild: GitHub Copilot's Cache Behavior at ScaleThis episode examines a production-scale characterization of GitHub Copilot's coding agent, built on sampled client-side telemetry from one week in June 2026 (13.5 million…
- Model Soups: Averaging Fine-Tuned Weights Beats Ensembles' CostThis episode examines "Model soups," a paper showing that averaging the weights of several fine-tuned models can raise accuracy without adding any inference cost. It explains why…
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory EfficientThis episode explores the paper "Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient," which asks whether a mixture-of-experts model can beat a dense model on loss…
- DigitalCoach: Gaps in Human and AI Software CoachingThis episode examines DIGITALCOACH, a dataset of 72 human coaching sessions (22,752 dialogue turns across Excel, FL Studio, Blender, Figma and Onshape) built to test whether…
- Soup-of-Experts: Pretraining Specialists via Parameter AveragingThis episode examines "Soup-of-Experts," an Apple paper on pretraining one bank of parameters that can be averaged into a small specialist model for any data mixture, with no…
- Hydragen: Shared-Prefix Attention for High-Throughput LLM InferenceThis episode examines Hydragen, a technique for speeding up LLM inference when many requests share a long prompt prefix, and asks how much of its claimed 32x end-to-end…
- Meta and AMD Co-Design MI350X for Recommendation Model TrainingThis episode examines a Meta and AMD paper claiming that AMD's MI350X can train Meta's production recommendation models at roughly the throughput of NVIDIA's B200. It explains…
- Context Language Models: Letting Models Edit Their Own ContextThis episode explores Context Language Models, a proposal that lets a model write directly to its own context as an editable file instead of relying on human-designed harnesses…
- 500xCompressor: Compressing Prompts into a Single Learned TokenThis episode examines "500xCompressor," a soft prompt compression method that squeezes hundreds of tokens of context into as little as a single learned token. A frozen Llama-3-8B…
- HyperTuning: Adapting Language Models Without Back-propagation Using HypernetworksThis episode examines "HyperTuning," an ICML 2023 paper that uses a hypernetwork (HyperT5) to adapt a frozen language model to a new task in a single forward pass, with no…
- Formalizing Hopfield Networks and Boltzmann Machines in Lean 4This episode explores a Lean 4 formalization of Hopfield networks and Boltzmann machines, covering about sixteen thousand new lines on top of mathlib, which the authors say is…
- HyperTuning via Gisting: Cheap Adapter Generation, Mixed ResultsThis episode examines "Investigating the Effectiveness of HyperTuning via Gisting," which asks whether a few-shot examples can be compressed into just sixteen tokens of state so…
- How Linear Attention Remembers: Writing and Recalling FactsThis episode examines "How Linear Attention Remembers," a preprint that asks how a single fact is written into, retained in, and read back out of the fixed-size recurrent state…
- Mixture of a Million Experts: Scaling Routing with Product KeysThis episode examines "Mixture of A Million Experts," a 2024 paper proposing a learned router that selects among more than a million single-neuron experts at sublinear cost. It…
- Gist Tokens: Compressing Prompts Up to 26xThis episode examines "Learning to Compress Prompts with Gist Tokens," which trains a language model to squeeze an instruction into a few cached gist-token activations. The trick…
- Scaling Laws for Fine-Grained Mixture of ExpertsThis episode examines a 2024 paper arguing that mixture-of-experts models outperform dense Transformers at every compute budget, with a gap that widens at scale. That claim…
- Compact Proofs of Model Performance via Mechanistic InterpretabilityThis episode explores a 2024 paper that tries to turn mechanistic interpretability into formal guarantees. The idea is to reverse-engineer a small transformer into an algorithm…
- Formal Lean Proof of PAC Learnability for Decision StumpsThis episode explores a 2021 formal proof, in the Lean theorem prover, that decision stumps (one-threshold classifiers) are PAC learnable. It starts from a simple learner that…
- Backprop as Functor: Composing Learners Through Category TheoryThis episode examines "Backprop as Functor," which argues that gradient descent on a whole network equals the composition of the gradient-descent learners of its parts. It…
- Verifying Peephole Rewrites in SSA Compiler IRs with LeanThis episode examines "Verifying Peephole Rewriting In SSA Compiler IRs," which uses the Lean proof assistant to give MLIR-style compilers a push-button workflow for proving…
- Exact Learning Dynamics in Deep Linear Neural NetworksThis episode explores the Saxe, McClelland, and Ganguli paper on exact learning dynamics in deep linear neural networks. It explains why these networks are worth studying even…
