HackerPapers

arXiv:2502.12115cs.LG

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke

Abstract

We introduce SWE-Lancer, a benchmark of over 1,400 freelance software engineering tasks from Upwork, valued at \$1 million USD total in real-world payouts. SWE-Lancer encompasses both independent engineering tasks--ranging from \$50 bug fixes to \$32,000 feature implementations--and managerial tasks, where models choose between technical implementation proposals. Independent tasks are graded with end-to-end tests triple-verified by experienced software engineers, while managerial decisions are assessed against the choices of the original hired engineering managers. We evaluate model performance and find that frontier models are still unable to solve the majority of tasks. To facilitate future research, we open-source a unified Docker image and a public evaluation split, SWE-Lancer Diamond (https://github.com/openai/SWELancer-Benchmark). By mapping model performance to monetary value, we hope SWE-Lancer enables greater research into the economic impact of AI model development.

arXiv comments
9 pages, 30 pages appendix
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.SESoftware Engineering

On Hacker News

Linked by 3 identifiable commenters in 4 comments across 4 threads, from Feb 2025 to May 2025. Found in 2 HN submissions from Feb 2025 to Feb 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Semantic unit testing: test code without executing it 1 comment

    • I feel this makes some fundamental conceptual mistakes and is just riding the LLM wave. "Semantics" is literally behavior under execution. This is syntactical analysis by a stochastic language model. I know the NLP literature uses "semantics" to talk about representations but…

      sigtstpView on HNParent context

  2. Past and Present Futures of User Interface Design 1 comment

    • This comment was honestly shocking. I spent more than 15 minutes trying to think of the first thing to bring up, but there were too many things to go through. I went on a tangent about belief bubbles then thought replying was too taxing, then I realised that not replying may be…

      spartanatreyuView on HNParent context

  3. GPT-4.5 1 comment

    • To put that in context, Claude 3.5 Sonnet (new), a model we have had for months now and which from all accounts seems to have been cheaper to train and is cheaper to use, is still ahead of GPT-4.5 at 36.1% vs 32.6% in SWE-Lancer Diamond [0]. The more I look into this release…

      TopfiView on HNParent context

  4. OpenAI Researchers Find That AI Is Unable to Solve Most Coding Problems 1 comment

    • Link to the original paper: https://arxiv.org/pdf/2502.12115 TL;DR: They tested with programming tasks and manager's tasks. The vast majority of tasks given require bugfixes. Claude 3.5 Sonnet (the best performing LLM) passed 21.1% of programmer tasks and 47.0% of manager tasks.…

      spartanatreyuView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.