HCAST: Human-Calibrated Autonomy Software Tasks — mentioned by ben_w

Mon, 22 Jun 2026 10:38:30 GMT

> Lousy benchmark Make your own then. It can go on the pile with all the others that keep getting saturated too fast to be useful. > they explicitly focus on the easiest tasks to automate for AI (i.e. heavily cherry picked outcomes) and it seems that they don't bother to test anything except just-released proprietary models. What? They made the benchmark last year, and included a bunch of models going back as far as 2019. When they first announced it, the top end of their tests were things AI could not actually automate, and even now only does erratically. Examples of the tasks SOTA models…

Paper and discussion history. A mention is not an endorsement.

HCAST: Human-Calibrated Autonomy Software Tasks — mentioned by yorwba

Sat, 16 May 2026 10:42:44 GMT

The METR task set contains no tasks with a duration greater than 32 hours (conservatively eyeballed from Figure 3: https://arxiv.org/abs/2503.17354 ), so any prediction that naively forecasts a longer time horizon is trivially incorrect. I guess that won't lead to a sigmoid-looking graph though, since METR will likely switch to a different evaluation methodology at that point and stop updating the old curve.

Paper and discussion history. A mention is not an endorsement.

HCAST: Human-Calibrated Autonomy Software Tasks · HackerPapers discussions

HCAST: Human-Calibrated Autonomy Software Tasks — mentioned by ben_w

HCAST: Human-Calibrated Autonomy Software Tasks — mentioned by yorwba