SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? · HackerPapers discussions

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — mentioned by sigtstp

Mon, 05 May 2025 08:17:15 GMT

I feel this makes some fundamental conceptual mistakes and is just riding the LLM wave. "Semantics" is literally behavior under execution. This is syntactical analysis by a stochastic language model. I know the NLP literature uses "semantics" to talk about representations but that is an assertion which is contested [1]. Coming back to testing, this implicitly relies on the strong assumption of the LLM correctly associating the code (syntax) with assertions of properties under execution (semantic properties). This is a very risky assumption considering, once again, these things are stochastic…

Paper and discussion history. A mention is not an endorsement.

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — mentioned by spartanatreyu

Wed, 19 Mar 2025 02:21:17 GMT

This comment was honestly shocking. I spent more than 15 minutes trying to think of the first thing to bring up, but there were too many things to go through. I went on a tangent about belief bubbles then thought replying was too taxing, then I realised that not replying may be contributing to the problem of those bubbles existing in the first place. So I'll try to keep my reply brief and succinct enough to not lose my mind by just responding to only some points you brought up. > Last year, many, probably most experienced programmers were skeptical about using LLMs to help write code. Firstly…

Paper and discussion history. A mention is not an endorsement.

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — mentioned by Topfi

Thu, 27 Feb 2025 20:38:34 GMT

To put that in context, Claude 3.5 Sonnet (new), a model we have had for months now and which from all accounts seems to have been cheaper to train and is cheaper to use, is still ahead of GPT-4.5 at 36.1% vs 32.6% in SWE-Lancer Diamond [0]. The more I look into this release, the more confused I get. [0] https://arxiv.org/pdf/2502.12115

Paper and discussion history. A mention is not an endorsement.

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — mentioned by spartanatreyu

Mon, 24 Feb 2025 05:35:30 GMT

Link to the original paper: https://arxiv.org/pdf/2502.12115 TL;DR: They tested with programming tasks and manager's tasks. The vast majority of tasks given require bugfixes. Claude 3.5 Sonnet (the best performing LLM) passed 21.1% of programmer tasks and 47.0% of manager tasks. The LLMs have a higher probability of passing the tests when they are given more attempts, but there's not a lot of data showing where the improvement tails off. (probably due to how expensive it is to run the tests) Personally, I have other concerns: - A human being asked to review repeated LLM attempts to resolve a…

Paper and discussion history. A mention is not an endorsement.

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — HN submission

Tue, 18 Feb 2025 05:40:39 GMT

SWE-Lancer: Can LLMs Earn $1M from Real-World Freelance Software Engineering?

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? — HN submission

Tue, 18 Feb 2025 05:25:05 GMT

SWE-Lancer: a benchmark of freelance software engineering tasks from Upwork

111 points; 74 discussion replies on HN. Paper and source history. A submission is not an endorsement.