SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration · HackerPapers discussions

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration — mentioned by mexicocitinluez

Fri, 09 Oct 2026 11:07:58 GMT

https://arxiv.org/abs/2603.03823

Paper and discussion history. A mention is not an endorsement.

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration — mentioned by keheliya

Mon, 22 Jun 2026 11:41:56 GMT

There are upcoming benchmarks aimed at measuring the ability to work with brownfield tasks. (Of course, benchmarks can be gamed, but they are still better than unrealistic toy tasks that earlier generations of benchmarks used. Frontier labs are yet to use them in their tech reports or marketing material, though.:-) * SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios https://arxiv.org/abs/2512.18470 * SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration https://arxiv.org/abs/2603.03823

Paper and discussion history. A mention is not an endorsement.

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration — mentioned by pron

Wed, 06 May 2026 14:04:12 GMT

Nah, it's a skill issue on the part of those who believe in "agent swarms" (in fact, that's how I recognise AI noobs; they think swarms work). Studies (like this [1]) and Anthropic's experiements have told us they don't. We do experiments with software correctness and formal methods experts who actually dive deep into "swarm outputs" and try to put evolutionary pressure on them. Swarms simply cannot (yet) produce viable software. They do, however, produce software that for a while passes tests. What I think is happening is that people who believe swarms work just look at test results. But…

Paper and discussion history. A mention is not an endorsement.

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration — mentioned by sanxiyn

Tue, 17 Mar 2026 01:15:35 GMT

The study is likely "SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration". Regression rate plot is figure 6. Read the study to understand what it is measuring and how it was measured. As I understand parent's summary is fine, but you want to understand it first before repeating it to others. https://arxiv.org/abs/2603.03823

Paper and discussion history. A mention is not an endorsement.

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration — HN submission

Sun, 08 Mar 2026 08:11:50 GMT

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

125 points; 41 discussion replies on HN. Paper and source history. A submission is not an endorsement.

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration — HN submission

Sat, 07 Mar 2026 21:04:04 GMT

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via CI

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.