SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks · HackerPapers discussions

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — mentioned by weiliddat

Tue, 28 Jul 2026 07:15:28 GMT

For most of these benchmarks, I feel like a p50 and p95 (using SWE salary as a proxy?) human benchmark as reference would be interesting. Edit: FWIW the paper the post quoted has repositories as slop baseline https://arxiv.org/html/2603.24755v1#S4.SS2

Paper and discussion history. A mention is not an endorsement.

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — mentioned by dhorthy

Tue, 28 Jul 2026 00:03:55 GMT

no i'm spinning those up at some point this week. here's the first few prompts I used (claude opus 5 as the research orchestrator), (these were interspersed with lots of tools and assistant messages but it should get you kicked off. > fetch this article for slopcodebench and help me run an eval on a subset of problems with opus 5 https://arxiv.org/html/2603.24755v1 > Get all the context, fetch any mentioned repos, and then propose a plan to me. > i have an anthropic API key in .... > Let's do the three challenges with Opus 4.8 and Opus 5 and Fable please. I like your minimal set. Let's try…

Paper and discussion history. A mention is not an endorsement.

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — mentioned by djx22

Mon, 13 Jul 2026 17:36:03 GMT

Actually, this paper shows that the LLMs create a lot more slop and degradation of the code after the first prompt. https://arxiv.org/html/2603.24755v1. So iterating with the LLM will simply make it to produce worse and worse code.

Paper and discussion history. A mention is not an endorsement.

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — mentioned by geraltofrivia

Mon, 06 Jul 2026 08:17:19 GMT

Our initial suggestion was to do something along the lines of your proposal. But we found that when we ask an agent to implement features for `t-n`^th PR, even when we are overly specific makes the code rather divergent, to the point that sometimes the `t-n+1`th task description doesn't really make a lot of sense. In fact, there are some papers (that we cited) which create a set of tasks doing exactly this, and it is non-trivial [1]. [1] https://arxiv.org/html/2603.24755

Paper and discussion history. A mention is not an endorsement.

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — HN submission

Sun, 05 Jul 2026 23:09:48 GMT

SlopCodeBench: Benchmarking How Coding Agents Degrade over Long, Iterative Tasks

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — mentioned by hirako2000

Wed, 13 May 2026 06:26:52 GMT

What feels already like old history is that Apple made a generous deal to OpenAI based on the premise that their AI could do the claims. Apple engineers spend months trying to prompt engineer their way, thinking the prompter is at fault if the soon to be AGI system diverged. Some of these instructions were trending out there, as reveals of how naive Apple was at the time. They could be traced from the device's logs so not so much of a leak: Don't hallucinate, strictly follow instructions, followed by all sort of refined predicates, appended as if an LLM had reason Then Apple released a paper…

Paper and discussion history. A mention is not an endorsement.

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — mentioned by hirako2000

Mon, 11 May 2026 04:51:57 GMT

Research also makes similar claims: https://arxiv.org/html/2603.24755v1

Paper and discussion history. A mention is not an endorsement.

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — HN submission

Tue, 07 Apr 2026 09:45:49 GMT

SlopCodeBench: Benchmarking How Coding Agents Degrade over Long-Horizon Task

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — mentioned by kayge

Tue, 31 Mar 2026 16:59:10 GMT

These links were previously submitted to HN without much traction: Research Paper — https://arxiv.org/abs/2603.24755 SlopCodeBench — https://www.scbench.ai/ And of course, the XCancel version of this post: https://xcancel.com/godofprompt/status/2038335755995750758

Paper and discussion history. A mention is not an endorsement.

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — mentioned by FiberBundle

Tue, 31 Mar 2026 10:22:18 GMT

This is the dumbest take there is about vibe coding. Claiming that managing complexity in a codebase doesn't matter anymore. I can't imagine that a competent engineer would come to the conclusion that managing complexity doesn't matter anymore. There is actually some evidence that coding agents struggle the same way humans do as the complexity of the system increases [0]. [0] https://arxiv.org/abs/2603.24755

Paper and discussion history. A mention is not an endorsement.

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks — HN submission

Fri, 27 Mar 2026 22:44:40 GMT

SlopCodeBench: Benchmarking How Coding Agents Degrade over Long-Horizon Tasks

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.