Gå til innholdet
Global sound
Linear Digressions

Linear Digressions

Distillation, or, How to Steal a Model

· 24 min

Episodebeskrivelse

This week we’re covering model distillation: the technique of using a large "teacher" model's outputs to train a smaller, cheaper "student" model that mimics it. They cover the two big reasons labs do this — making lighter, faster, more focused models for specific tasks, and the more contentious use case of effectively copying a rival's flagship model by hammering its API with questions (with a callback to the old Bing/Google search controversy). They also get into why it's so hard to prove distillation happened, why some models occasionally introduce themselves as "Claude," and a surprisingly old idea: a 2015 paper by Geoffrey Hinton, Jeff Dean, and Oriol Vinyals on distilling knowledge using the full probability distribution over a model's outputs — not just its single most likely answer — and what that "soft label" approach captures about how a model relates concepts to each other.

Nettsted: soundcloud.com

Live, siste time: 11 347 låter på 5 883 stasjoner i 142 land

Oppdag verden gjennom radioen

Flere måter å oppdage på (9)

Hørt noe fint? Lagre det, eller åpne det rett i Spotify.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Magic Viva ItaliaFortsett å lytte