Show notes
You trust your AI coding agent to understand what its code does once it runs. A new benchmark says that trust is misplaced most of the time.
Researchers built SWE-Flux to test it: 480 questions across 12 real Python repos, every answer checked against an actual test run. Not a guess.
The questions aren't "what does this code do." They're "what happens when it runs." Which value moves where. Which branch a loop takes. Why an exception fires.
The best of five models got 37% right. Not on the hardest questions. Overall.
Give them one function. A simple loop, a local check. They're fine.
Ask…
Website: rss.com
