Ir para o conteúdo
Global sound
Daily Paper Cast

Daily Paper Cast

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

· 25 min · Episódio 2506

Notas do episódio

🤗 Upvotes: 27 | cs.SE, cs.AI

Authors:

Xinping Lei, Minghao Liu, Ruijie Zhu, Xiaohui Hu, Xingqiao Li, Ao Zhang, Wenhao Chai, Han Li, Tao Wang, Jingyuan Zhang, Enduo Zhao, Yunfeng Shi, Yiming Liang, Zetao Yang, Xingjian Wang, Yifan Yao, Xinkai Ma, Yingru Li, Tianhan Yang, Huan Zhou, Weiran Shi, Shen Yan, Jiaheng Liu, Stephen Huang, Eli Cheung, Weihao Xuan

Title:

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

Arxiv:

http://arxiv.org/abs/2611.20218v1

Abstract:

A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. We evaluate eleven models and find a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.

Site: share.transistor.fm

Ao vivo, última hora: 11.352 músicas em 5.429 rádios de 146 países

Descubra o mundo pelo rádio

Mais formas de descobrir (7)

Ouviu algo legal? Salve ou abra direto no Spotify.

★ Artista da semana A maior subida na rádio esta semana
A tua música aqui? Desde € 9 por semana →
Radio TropiexitosContinuar ouvindo