RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts — mentioned by namaria

Tue, 22 Apr 2025 11:48:16 GMT

As usual with these bombastic blog posts, you have to follow the paper trail to find the caveats that deflate the whole thing. From the papers about the tasks used in this estimation, we can easily find out that: LLMs never exceed 20% success rate on tasks taking more than 4h: HCAST: Human-Calibrated Autonomy Software Tasks https://metr.org/hcast.pdf LLMs hit a score ceiling at around 1-2h mark; LLM performance drops off steeply with increase in LOC count, never breaching 50% success rate after 800 lines and below 40% at the longest one at ~1600 lines. RE-Bench: Evaluating frontier AI R&D…

Paper and discussion history. A mention is not an endorsement.

RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts — HN submission

Tue, 31 Dec 2024 20:19:14 GMT

Re-Bench: Evaluating ML agents against human ML experts

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts · HackerPapers discussions

RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts — mentioned by namaria

RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts — HN submission