Skip to content
Global sound
Snacks Weekly on Data Science

Snacks Weekly on Data Science

Evaluating Show Synopses with LLM-as-a-Judge [Netflix]

· 12 min · Episode 148

Show notes

In this episode, we discuss how Netflix built an LLM-based evaluation framework for creative writing quality without losing the nuance that makes expert review valuable. They achieved this by creating a golden dataset calibrated by expert writers, splitting evaluation into specialized judges for each quality criterion, and applying targeted techniques—tiered reasoning, consensus sampling, and multi-agent verification—tailored to the unique needs of each criterion.
For more details, you can refer to their published tech blog, linked here for your reference…

Website: podcasters.spotify.com

Live, last hour: 15,793 songs on 6,517 stations in 146 countries

Discover the world through radio

More ways to discover (9)

Heard something good? Save it, or open it straight away in Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Radio NeulandContinue listening