SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? · HackerPapers discussions

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? — mentioned by djoldman

Sun, 26 Apr 2026 15:17:00 GMT

> We have incorporated these findings into our recent evaluation efforts. In the last months we’ve chosen to report results from the public split of SWE-Bench Pro. We recommend other model developers do the same. SWE-bench Pro is not perfect, but empirically seems to suffer less from contamination issues. https://arxiv.org/pdf/2509.16941

Paper and discussion history. A mention is not an endorsement.

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? — mentioned by yorwba

Tue, 03 Feb 2026 17:04:50 GMT

SWE-Bench Pro consists of 1865 tasks. https://arxiv.org/abs/2509.16941 Qwen3-Coder-Next solved 44.3% (826 or 827) of these tasks. To solve a single task, it took between ≈50 and ≈280 agent turns, ≈150 on average. In other words, a single pass through the dataset took ≈280000 agent turns. Kimi-K2.5 solved ≈84 fewer tasks, but also only took about a third as many agent turns.

Paper and discussion history. A mention is not an endorsement.