Data Science in the Wild
by Peter Liu
Latest episode
Ashoka University — When covariates hide the pre-trend
Ashoka University — When covariates hide the pre-trend
About this podcast
How do top tech companies actually ship machine learning? Each episode is a conversational deep dive into one real production system or research result, from teams like Netflix, Airbnb, Meta, Uber, Spotify, Instacart, Shopify, and Google. We cover the business problem, the technical approach, the trade-offs, and the measured impact.
Topics include A/B testing and experimentation, recommender systems and ranking, LLM evaluation, AI agents, search and retrieval, and applied causal inference. It's data science in the wild: not textbook theory, but the models and decisions real companies run in production. New episodes several times a week, about 15–20 minutes each, so they fit a commute.
Hosted by Peter Liu, a data scientist with 10+ years in consumer tech (Meta, Webflow, Adobe, eBay). Episodes are produced with AI-generated voices from curated source articles and papers, which are linked in each episode's notes.
Episodes
The newest 20 of 54 episodes
- Ashoka University — When covariates hide the pre-trend · 18 min · Season 2 Episode 21Do late covariates hide pre-trends? Arora (Ashoka) finds 77 of 90 DiD studies date a covariate too late—and honest bands can cover in only 5% of Monte Carlos. Key takeaways: •…
- Huawei — Does the Latest Search Actually Drive Recommendations? · 20 min · Season 2 Episode 20Do recommenders use your latest search? Huawei's FocalSwap audit of 66,139 rows finds the credit flips with the control: masking credits search, matched swaps don't. Key…
- University of Granada — Do LLM Recommenders Pick by Position? · 22 min · Season 2 Episode 19Do LLM recommenders pick by position? A University of Granada study ran 40,000 shuffled prompts through 8 open models, and every one failed both statistical tests for position…
- Independent research — Should You Recalibrate CTR Models on Fresh Data? · 21 min · Season 2 Episode 18Should you recalibrate CTR models on fresh data? Rui Liu's Avazu study: a 6-hour Platt/isotonic refit made calibration worse in about a third of applications. Key takeaways: •…
- Spotify — Can You Ship an LLM Change on Evals Alone? · 20 min · Season 2 Episode 17Can you ship an LLM change on evals alone? Spotify's Mårten Schultzberg: only if eval effects predict A/B results. On 183 Upworthy tests, his gate ships none. Key takeaways: •…
- Brown University — Is Difference-in-Differences Actually Broken? · 24 min · Season 2 Episode 16Is difference-in-differences broken? Brown economist Peter Hull used AI agents to replicate 1,346 staggered-adoption results from 108 papers: TWFE and modern DiD estimates…
- iFood — Predicting the Discount That Tips a Customer · 17 min · Season 2 Episode 15LCM-Promo (Prosus and iFood) turns a 4B-parameter LLM into an uplift model for discounts. In a one-month A/B test on churned users it lifted conversion 1% and cut promotion cost…
- Recommender Systems — How User Activity Creates Phantom Item Similarity · 18 min · Season 2 Episode 14Phantom correlation (Favier et al., University of Antwerp): user activity alone makes items look similar. On MovieLens-1M, an activity-only model removes 81% of the independence…
- Time-Series Foundation Models — Keeping Forecast Intervals Honest When Volatility Shifts · 19 min · Season 2 Episode 13Volatility-Corrected Prediction Intervals (Cho & Matteson, Cornell): time-series foundation models can hit 80% coverage on average yet miss when volatility rises. A GARCH…
- AI Analysts — Keeping Type I Error Under Control as They Evolve · 18 min · Season 2 Episode 12Lifetime-valid decisions by continually evolving AI analysts (Sun, Zeng & Wang, Purdue): every prompt edit or model swap gives an AI analyst another chance at a false positive…
- Bayesian LLM Agents — Reasoning Does the Math, Code Handles the Rest · 18 min · Season 2 Episode 11From Prompts to Posteriors (Filbry, Melo & Gal, Oxford): LLMs break Bayes' rule without reasoning, but reasoning models compute simple posteriors almost exactly, and with code…
- Instagram Feed — Keeping the Baseline in Ranking · 23 min · Season 2 Episode 10Instagram Feed's ranker stops deleting "bias": Meta keeps the raw engagement prediction and adds its debiased, incremental part as a second, separately weighted term. Key…
- Selling in the Wild — A Randomized Trial of an AI Sales Agent · 21 min · Season 2 Episode 9Selling in the Wild: in a 14.8M-visitor field trial, the Javo AI sales agent lifted revenue per visitor 10% and held its own against human chat staff at 1.8% of the cost. Key…
- Conformal Policy Selection — Safety Gates for Individual AI Decisions · 20 min · Season 2 Episode 8Conformal Policy Selection (Cao, Gao, Saar-Tsechansky & Yin) gates AI decisions person by person—zero harmful EHR treatments deployed for q ≤ 0.85. Key takeaways: • Per unit…
- Causal Forests — Treatment Effects on Zero-Heavy Outcomes · 27 min · Season 2 Episode 7ZIC-BCF-Smear (Souto, Wang & Shuryak) pairs two Bayesian causal forests for zero-heavy outcomes—finding panitumumab adds ~10 severe-toxicity days per patient. Key takeaways: •…
- WeChat — Generative Retrieval for Industrial Search · 25 min · Season 2 Episode 6WeChat's WeGR turns e-commerce search into generative retrieval: CoCo-VAE builds collaborative, collapse-free Semantic IDs, and DHPO trains with hierarchical partial…
- Uber — An AI First-Pass Reviewer for Product Requirement Docs · 21 min · Season 2 Episode 5Uber's PRD Evaluator is an AI first-pass reviewer that assembles company context, calibrates review depth by risk, and returns a scorecard with write-ready fixes—before human…
- ByteDance — OneTrans-V2: One Transformer for the Whole Recsys Cascade · 21 min · Season 2 Episode 4ByteDance's OneTrans-V2 runs retrieval, pre-rank and fine-rank as one Transformer: +9.74% GMV per user and 3.2× QPS on the same hardware in a 50/50 A/B test. Key takeaways: •…
- Meta — Expected Bayesian Loss for Neutral Launches · 22 min · Season 2 Episode 3Meta's Expected Bayesian Loss turns neutral-launch safety from a CI-overlap yes/no into a tunable downside-risk number, matched to experts on 17 experiments. Key takeaways: • CI…
- Etsy — Using Post-Treatment Data to Sharpen A/B Tests · 21 min · Season 2 Episode 2Etsy and UC Berkeley stack 23 screened in-experiment covariates on top of CUPAC, cutting A/B-test variance a further ~2.5–14% across 29 experiments. Key takeaways: • Two stages…
