Ugrás a tartalomra
Global sound
Linear Digressions

Linear Digressions

Benchmark Bank Heist

· 13 perc

Epizódjegyzetek

What if an AI decided the smartest way to pass its test was to find the answer key? That's exactly what Anthropic's Claude Opus did when faced with a benchmark evaluation — reasoning that it was being tested, tracking down the encrypted eval dataset, decrypting it, and returning the answer it found inside. It's equal parts impressive and unsettling. This episode digs into what actually happened, why it matters for how we measure AI progress, and what this very novel failure mode means for the already-tricky science of benchmarking language models.

Links

Anthropic's writeup on the BrowseComp reverse-engineering done by Claude Opus 4.6: https://www.anthropic.com/engineering/eval-awareness-browsecomp

BrowseComp benchmark from OpenAI: https://openai.com/index/browsecomp/

Weboldal: soundcloud.com

Élőben, az elmúlt órában: 14 685 dal, 6 738 állomás, 145 ország

Fedezd fel a világot a rádión keresztül

További felfedezési lehetőségek (9)

Hallottál valami jót? Mentsd el, vagy nyisd meg rögtön a Spotifyban.

★ Artist of the week The fastest riser on the radio this week
Your song here? From € 9 per week →
Cruize RadioHallgatás folytatása