Show notes
This episode examines "Intelligence per Watt," a Stanford and Together AI paper that asks whether small local models on laptop-class hardware can take real query load off the cloud. It walks through the paper's definitions. "Local" means at most 20 billion active parameters per token, which includes mixture-of-experts models like gpt-oss-120B. The proposed metric is average task accuracy divided by average power draw, and the paper also reports an energy-per-query (joules) variant. The discussion points out that watts alone ignore latency, so a low-power device that runs slowly can still cost more energy per query. It also flags the measurement caveats: power is read from the accelerator only, not the wall; queries run at batch size 1; chat accuracy is judged against a single reference model's answers; and "best-of-local" coverage is an oracle router that couldn't be deployed. The study spans about a million single-turn queries, more than twenty models and eight accelerators, and the episode places it against earlier LLM energy work such as "From Words to Watts" and MLPerf Power. Listeners get a careful guide to what the metric does and doesn't capture before the headline efficiency numbers are discussed.
Sources:
1. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, Shang Zhu, Ben Athiwaratkun, John Hennessy, Azalia Mirhoseini, Christopher Ré, 2025
http://arxiv.org/abs/2511.07885
2. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI — Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin, et al. (Stanford, Together AI; senior authors incl. John Hennessy, Azalia Mirhoseini, Christopher Ré), 2025 (v7 revised Oct 2026)
https://scholar.google.com/scholar?q=Intelligence+per+Watt%3A+Measuring+Intelligence+Efficiency+of+Local+AI
3. MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI — Arya Tschand, Vijay Janapa Reddi, et al. (MLCommons), 2024
https://scholar.google.com/scholar?q=MLPerf+Power%3A+Benchmarking+the+Energy+Efficiency+of+Machine+Learning+Systems+from+Microwatts+to+Megawatts+for+Sustainable+AI
4. Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model — Alexandra Sasha Luccioni, Sylvain Viguier, Anne-Laure Ligozat, 2022
https://scholar.google.com/scholar?q=Estimating+the+Carbon+Footprint+of+BLOOM%2C+a+176B+Parameter+Language+Model
5. From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inference — Siddharth Samsi, Dan Zhao, Joseph McDonald, et al. (MIT Lincoln Laboratory), 2023
https://scholar.google.com/scholar?q=From+Words+to+Watts%3A+Benchmarking+the+Energy+Costs+of+Large+Language+Model+Inference
6. Small Language Models are the Future of Agentic AI — Peter Belcak, Greg Heinrich, Shizhe Diao, et al. (NVIDIA…
