Vai al contenuto
Global sound
Data Science in the Wild

Data Science in the Wild

Spotify — Can You Ship an LLM Change on Evals Alone?

· 20 min · Stagione 2 Episodio 17

Note dell’episodio

Can you ship an LLM change on evals alone? Spotify's Mårten Schultzberg: only if eval effects predict A/B results. On 183 Upworthy tests, his gate ships none.
Key takeaways:

• The question. Judge–human agreement doesn't tell you which changes can skip an A/B test. That needs trial-level surrogacy: eval treatment effects that predict online effects across a class of interventions. It's logically independent of unit-level (Prentice) surrogacy.
• The gate. Subtracting each intervention's reported noise variances from the covariance undoes attenuation (a random-effects correction dating to…

Sito web: rss.com

In diretta, ultima ora: 13.377 brani su 5.476 stazioni in 117 paesi

Scopri il mondo attraverso la radio

Altri modi per scoprire (7)

Hai sentito qualcosa di bello? Salvalo, o aprilo subito in Spotify.

★ Artista della settimana Il brano che sale più in fretta in radio questa settimana
Il tuo brano qui? Da € 9 a settimana →
Kral Türk FMContinua ad ascoltare