Notatki do odcinka
To address issues of pretraining contamination and brittle inherited grading in software engineering benchmarks, researchers introduced DeepSWE, an evaluation suite featuring original, long-horizon coding tasks authored entirely from scratch across 91 open-source repositories. Unlike traditional benchmarks that mine merged pull requests and rely on tests written for specific historical fixes, DeepSWE employs functional verifiers that check observable software behavior and accept any valid implementation of the requested features. Experiments demonstrate that this approach separates frontier…
Strona: podcasters.spotify.com

