Tietoa podcastista
An automatically generated podcast about machine learning and natural language processing. The two fictional hosts talk about papers that I want to learn more about on my way to work. It's not good, but it's useful.
Jaksot
Uusimmat 20 jaksoa 170 jaksosta
- Qwen3.8-Next · 22 minIn this episode: • Welcome and the 1/9th FLOP Miracle: Linda introduces Qwen3.8-Next and its massive efficiency gains, hooking Dr. Norris. • Token Mixing with GDN and Sparse…
- Distillation Scaling Laws · 19 minIn this episode: • Introduction to Distillation and Inference Costs: Linda and Norris introduce the paper and discuss why compute-optimal models are driving the need for smaller…
- Decode-Branch Transformers · 20 minIn this episode: • The Inference Bottleneck: Introduction to the podcast and the fundamental asymmetry between compute-bound prefill and memory-bound decoding in LLMs. • Enter…
- The State-Prediction Separation Hypothesis · 19 minIn this episode: • Introduction to State-Prediction Separation: Professor Norris and Linda introduce the paper and the core problem of standard Transformers conflating next-token…
- Pretraining Mixture of Experts for Emergent Modularity · 22 minIn this episode: • Introduction to Monolithic MoEs: Linda introduces EMO and Professor Norris discusses the memory bottlenecks of standard monolithic Large Language Models. • The…
- Muon-Optimized Distillation and Quantization for LLM Deployment · 22 minIn this episode: • Introduction & The Edge Deployment Problem: Professor Norris and Linda introduce the episode's paper, focusing on the difficulty of running large language…
- Continuous-Query Limited Memory Language Models · 21 minIn this episode: • Welcome & The Parametric Memory Bottleneck: Professor Norris and Linda introduce the episode's topic, discussing the fundamental flaws of storing facts in…
- QuaRot · 22 minIn this episode: • Introduction to Quantization and Outliers: Norris and Linda introduce the episode's paper, QuaRot, and discuss the main hurdle in LLM inference: the memory…
- How to set AdamW’s weight decay as you scale model and dataset size · 22 minIn this episode: • Introduction to Mechanical Dreams and the AdamW Scaling Problem: Professor Norris and Linda introduce the podcast and discuss the costly problem of…
- Weight Decay Improves Language Model Plasticity · 20 minIn this episode: • Introduction and The Premise: Professor Norris and Linda introduce the episode's focus: a new paper showing how weight decay impacts language model plasticity…
- Bridging Compute- and Data-Optimal Pretraining · 20 minIn this episode: • The Data Wall: Introduction to the problem of running out of fresh high-quality pretraining data and the limitations of classical Chinchilla scaling. • The…
- Pushing LLM Pretraining Scales · 22 minIn this episode: • Introduction to Higher-Order Optimizers: Linda introduces the NVIDIA paper on SOAP and Muon, and Professor Norris expresses his initial skepticism about…
- Why Larger Models Learn More · 23 minIn this episode: • Introduction: The Mystery of Scaling: Professor Norris and Linda introduce the core question of why larger models learn tasks that smaller ones cannot…
- Rethinking the Role of Efficient Attention in Hybrid Architectures · 21 minIn this episode: • Introduction to Hybrid Architectures: Professor Norris and Linda introduce the episode's paper on hybrid attention architectures and discuss why scaling full…
- Nemotron-Labs-Diffusion · 21 minIn this episode: • Welcome and the Autoregressive Bottleneck: Linda introduces the Nemotron-Labs-Diffusion paper and the hosts discuss the fundamental limits of sequential…
- Lightning Index Ultra-Long Context via Lookahead Sparse Attention · 25 minIn this episode: • The Memory Wall and the 90 Percent Waste: Professor Norris and Linda introduce the episode's paper, discussing the GPU memory bottleneck caused by ultra-long…
- Learning beyond Teacher · 21 minIn this episode: • Distillation vs. Reinforcement Learning: Linda and Norris introduce the paper and discuss the differences between off-policy and on-policy distillation. • The…
- Value Residual Learning · 22 minIn this episode: • Welcome and the Over-smoothing Problem: Professor Norris and Linda introduce the episode and discuss how deep Transformers suffer from over-smoothing, losing…
- Learning Rates Regulate Catastrophic Overtraining · 21 minIn this episode: • Introduction to Catastrophic Overtraining: Linda and Professor Norris introduce the paper and the counterintuitive phenomenon where better pretraining leads to…
- HRM-Text · 22 minIn this episode: • Introduction to the Compute Divide: Linda and Professor Norris introduce the podcast and discuss the massive computational barriers in modern LLM pretraining…
