Note dell’episodio
Can you ship an LLM change on evals alone? Spotify's Mårten Schultzberg: only if eval effects predict A/B results. On 183 Upworthy tests, his gate ships none.
Key takeaways:
• The question. Judge–human agreement doesn't tell you which changes can skip an A/B test. That needs trial-level surrogacy: eval treatment effects that predict online effects across a class of interventions. It's logically independent of unit-level (Prentice) surrogacy.
• The gate. Subtracting each intervention's reported noise variances from the covariance undoes attenuation (a random-effects correction dating to…
Sito web: rss.com

