SWE-bench: Can Language Models Resolve Real-World GitHub Issues? · HackerPapers discussions

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — mentioned by chaostheory

Wed, 07 Oct 2026 05:32:08 GMT

> Prove it. Anyone can make up a anecdote. In my case an agent was able to do a task in 10 min vs almost an hour without documentation. Citing an anecdote as “proof” isn’t good enough. Here is my proof: Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024). On the Impacts of Contexts on Repository-Level Code Generation. arXiv. https://doi.org/10.48550/arxiv.2406.11927 Cited by: 46 Jimenez, C. E., Yang, J., Wettig, A., et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? arXiv. https://doi.org/10.48550/arxiv.2310.06770 Cited by: 4998 Zhang, F., Chen, B., Zhang, Y., et…

Paper and discussion history. A mention is not an endorsement.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — mentioned by ahmed-fathi

Thu, 12 Mar 2026 21:43:46 GMT

No single paper nails that exact claim. SWE-bench Princeton does show that models struggle significantly with real-world issues requiring changes across multiple files and functions which points in that direction. But the local vs global framing is mostly practitioner-observed, not a formally tested hypothesis yet. Fair point, I should have hedged it. https://arxiv.org/abs/2310.06770

Paper and discussion history. A mention is not an endorsement.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — mentioned by kristianp

Wed, 10 Dec 2025 20:51:08 GMT

I know python is more prevalent in SWE-Bench than any other language, but more than 50% django sounds like a big stretch. Citation? Edit, it's about 37%, and python-only. https://arxiv.org/pdf/2310.06770v3

Paper and discussion history. A mention is not an endorsement.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — mentioned by sigmoid10

Fri, 12 Sep 2025 15:11:38 GMT

You apparently don't know what SWE-bench is [1]. First of all, it tries to evaluate skills that explicitly go beyond blank project questions with given solutions. Secondly, it does not contain "optimal" or sometimes even correct solutions. That's because it uses real world software development examples from actual PRs in popular repos. These very likely had humans use all the tools at their disposal as well (e.g. web search, git commands, code execution). Assuming an LLM could have solved these just by looking at a piece of code turns out to be very myopic. [1]…

Paper and discussion history. A mention is not an endorsement.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — HN submission

Wed, 07 Feb 2024 20:55:08 GMT

SWE-BENCH: Can language models resolve real-world GitHub issues?

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — mentioned by belter

Wed, 31 Jan 2024 15:59:27 GMT

"SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" - https://arxiv.org/abs/2310.06770 "... Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. Claude 2 and GPT-4 solve a mere 4.8% and 1.7% of instances respectively, even when provided with an oracle retriever..."

Paper and discussion history. A mention is not an endorsement.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — HN submission

Thu, 12 Oct 2023 20:18:26 GMT

SWE-Bench: Can Language Models Resolve Real-World GitHub Issues?

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.