HackerPapers

arXiv:2310.06770cs.CL

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan

Abstract

Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of $2,294$ software engineering problems drawn from real GitHub issues and corresponding pull requests across $12$ popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere $1.96$% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.

arXiv comments
Data, code, and leaderboard are available at https://www.swebench.com ICLR 2024, https://openreview.net/forum?id=VTF8yNQM66
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.SESoftware Engineering

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Jan 2024 to Oct 2026. Found in 2 HN submissions from Oct 2023 to Feb 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Agents don't need memory, they need documentation 1 comment

    • > Prove it. Anyone can make up a anecdote. In my case an agent was able to do a task in 10 min vs almost an hour without documentation. Citing an anecdote as “proof” isn’t good enough. Here is my proof: Hai, N. L., Nguyen, D. M., & Bui, N. D. Q. (2024). On the Impacts of…

      chaostheoryView on HNParent context

  2. Ask HN: Does code style matter much anymore? 1 comment

    • No single paper nails that exact claim. SWE-bench Princeton does show that models struggle significantly with real-world issues requiring changes across multiple files and functions which points in that direction. But the local vs global framing is mostly practitioner-observed…

      ahmed-fathiView on HNParent context

  3. Django: what’s new in 6.0 1 comment

    • I know python is more prevalent in SWE-Bench than any other language, but more than 50% django sounds like a big stretch. Citation? Edit, it's about 37%, and python-only. https://arxiv.org/pdf/2310.06770v3

      kristianpView on HNParent context

  4. Top model scores may be skewed by Git history leaks in SWE-bench 1 comment

    • You apparently don't know what SWE-bench is [1]. First of all, it tries to evaluate skills that explicitly go beyond blank project questions with given solutions. Secondly, it does not contain "optimal" or sometimes even correct solutions. That's because it uses real world…

      sigmoid10View on HNParent context

  5. Researchers say generative AI isn't replacing devs any time soon 1 comment

    • "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" - https://arxiv.org/abs/2310.06770 "... Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. Claude 2 and GPT-4 solve a mere…

      belterView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.