The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason · HackerPapers discussions

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason — mentioned by Topfi

Sat, 06 Jun 2026 11:01:33 GMT

> […] fed large chunks of an article multiple times to an LLM […] So they had to prompt? An LLM? I got this argument before and still don’t get what it’s trying to say. These models do not output anything unless prompted, that’s not any kind of gotcha. On the code outputting front there is a lot of relevant evidence beyond the NYC lawsuit [0]. If I slightly modify GPL code, that doesn’t give me the right to relicense. [0] https://arxiv.org/html/2601.02671?amp=&amp= and https://arxiv.org/abs/2506.12286 and https://ai.stanford.edu/blog/verbatim-memorization/

Paper and discussion history. A mention is not an endorsement.

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason — mentioned by Topfi

Fri, 20 Mar 2026 19:23:11 GMT

> Modern LLMs don't memorize harry potter like GPT3 did. [...] You can't reliably do this with Opus or GPT5. If you try any modern LLM, you will find that you can. Easily [0], reliably [1], consistently [2]. All these examples are with models released in 2025/26. [0] https://arxiv.org/html/2601.02671?amp=&amp= [1] https://arxiv.org/abs/2506.12286 [2] https://ai.stanford.edu/blog/verbatim-memorization/

Paper and discussion history. A mention is not an endorsement.

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason — HN submission

Mon, 02 Feb 2026 17:49:10 GMT

The SWE-Bench Illusion: When LLMs Remember Instead of Reason

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason — HN submission

Mon, 02 Feb 2026 17:41:25 GMT

Prediction: Claude 5 will be a major regression

At this point it should be completely obvious to everyone that there’s what is approximately a linear relationship between model cost and model performance. Anthropic is claiming that Claude 5 Sonnet will cost about half as much as their current SOTA models. Therefore, expect about half the performance. This is Anthropic’s version of GPT-5, i.e. a way to fool their customers into using a less compute intensive model, almost purely for the benefit of the company. But as usual, they will rig the benchmarks and make it appear as though the model is better at certain things, like coding. It’s an…

3 points; 12 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason — HN submission

Sat, 13 Sep 2025 17:34:01 GMT

The SWE-Bench Illusion: When LLMs Remember Instead of Reason

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason — mentioned by piskov

Thu, 11 Sep 2025 18:50:58 GMT

Not “may be”: just look how swe-bench scores drop to single digits once it in C# https://arxiv.org/html/2506.12286v3

Paper and discussion history. A mention is not an endorsement.

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason — mentioned by piskov

Mon, 08 Sep 2025 12:44:52 GMT

Check swe-bench results but for C#. It’s literally pathetic how these things just memorize, not achieve any actual problem-solving https://arxiv.org/html/2506.12286v3

Paper and discussion history. A mention is not an endorsement.

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason — mentioned by lossolo

Wed, 06 Aug 2025 21:10:04 GMT

12.8 GB is around 110 Gbits. Even at 4.25 bits/weight the network stores ~26 billion "micro weights". A 1,4k token space invaders snippet occupies ~1.1 kb compressed, the model could parametrize thousands of such snippets and still have more than 99% of its capacity left. This paper about LLM memorization is interesting, if you would to know more: https://arxiv.org/abs/2312.11658 and another recent interesting paper SWE bench illusion shows SOTA code LLM results collapsing once memorised github issues are filtered out: https://arxiv.org/pdf/2506.12286v1 Add to this that the common crawl…

Paper and discussion history. A mention is not an endorsement.

The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason — mentioned by criemen

Tue, 29 Jul 2025 22:30:12 GMT

Running evals aren't the problem, the problem is acquiring or building a high-quality, non-contaminated dataset. https://arxiv.org/abs/2506.12286 makes a very compelling case that swebench (and in extension, anything that's based on public source code) is most likely overestimating your agents actual capabilities.

Paper and discussion history. A mention is not an endorsement.