Measuring AI Ability to Complete Long Software Tasks · HackerPapers discussions

Measuring AI Ability to Complete Long Software Tasks — mentioned by Leynos

Fri, 15 May 2026 19:39:34 GMT

Look at the tasks in the benchmark (see §2 https://arxiv.org/html/2503.14499v3)

Paper and discussion history. A mention is not an endorsement.

Measuring AI Ability to Complete Long Software Tasks — mentioned by ben_w

Tue, 05 May 2026 17:43:15 GMT

> Not misunderstanding. Then why did you write "Also, it’s super easy to game. Insert random lags, reduce tokens/sec, there you have a model that maintains attention over “long-time horizons”"? The wall-clock time the LLM spends per task isn't the metric. How long you can leave the LLM alone, wall-clock time, without intervention, isn't "long-time horizons", it's more like "I gave it a list of tasks and it worked through them". Which is neat when it works, but different. > All I see now is celebration of how agents run for hours and handle “long-time horizons.” Yes? And? The long time…

Paper and discussion history. A mention is not an endorsement.

Measuring AI Ability to Complete Long Software Tasks — mentioned by JohnLocke4

Thu, 23 Apr 2026 19:27:41 GMT

Sometimes I wonder if innovation in the AI space has stalled and recent progress is just a product of increased compute. Competence is increasing exponentially[1] but I guess it doesn't rule it out completely. I would postulate that a radical architecture shift is needed for the singularity though [1]https://arxiv.org/html/2503.14499v1 *Source is from March 2025 so make of it what you will.

Paper and discussion history. A mention is not an endorsement.

Measuring AI Ability to Complete Long Software Tasks — mentioned by ben_w

Tue, 14 Oct 2025 22:19:55 GMT

Sure, but also the METR study showed the rate of change is t doubles every 7 months where t ~= «duration of human time needed to complete a task, such that SOTA AI can complete same with 50% success»: https://arxiv.org/pdf/2503.14499 I don't know how long that exponential will continue for, and I have my suspicions that it stops before week-long tasks, but that's the trend-line we're on.

Paper and discussion history. A mention is not an endorsement.

Measuring AI Ability to Complete Long Software Tasks — HN submission

Tue, 08 Jul 2025 03:44:36 GMT

Measuring AI Ability to Complete Long Tasks

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Measuring AI Ability to Complete Long Software Tasks — mentioned by falcor84

Tue, 13 May 2025 09:55:19 GMT

Well, if you were to ask me, I would say that the biggest bottle-neck is the productive time horizon for AIs, which is currently at about 1h before SoTA agents go off-rails, based on the METR report[0]. As the report models, this time horizon doubles approximately every 7 months (R² = 0.98 on a log-scale), and thus if we assume that a CEO's work requires a time horizon of e.g. 10,000 hours to effectively plan and implement a long-term strategy, then we could expect companies to be able to replace CEOs by around 2033. There are of course other factors at play, but knowing companies' propensity…

Paper and discussion history. A mention is not an endorsement.

Measuring AI Ability to Complete Long Software Tasks — mentioned by namaria

Tue, 22 Apr 2025 11:48:16 GMT

As usual with these bombastic blog posts, you have to follow the paper trail to find the caveats that deflate the whole thing. From the papers about the tasks used in this estimation, we can easily find out that: LLMs never exceed 20% success rate on tasks taking more than 4h: HCAST: Human-Calibrated Autonomy Software Tasks https://metr.org/hcast.pdf LLMs hit a score ceiling at around 1-2h mark; LLM performance drops off steeply with increase in LOC count, never breaching 50% success rate after 800 lines and below 40% at the longest one at ~1600 lines. RE-Bench: Evaluating frontier AI R&D…

Paper and discussion history. A mention is not an endorsement.

Measuring AI Ability to Complete Long Software Tasks — HN submission

Thu, 03 Apr 2025 14:03:58 GMT

Measuring AI Ability to Complete Long Tasks

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Measuring AI Ability to Complete Long Software Tasks — HN submission

Fri, 21 Mar 2025 10:16:49 GMT

Measuring AI Ability to Complete Long Tasks

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Measuring AI Ability to Complete Long Software Tasks — HN submission

Thu, 20 Mar 2025 12:15:14 GMT

Measuring AI Ability to Complete Long Tasks

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Measuring AI Ability to Complete Long Software Tasks — HN submission

Wed, 19 Mar 2025 21:18:19 GMT

Measuring AI Ability to Complete Long Tasks

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.