H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark · HackerPapers discussions

H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark — mentioned by hypoxia

Fri, 20 Dec 2024 20:39:08 GMT

Many are incorrectly citing 85% as human-level performance. 85% is just the (semi-arbitrary) threshold for the winning the prize. o3 actually beats the human average by a wide margin: 64.2% for humans vs. 82.8%+ for o3. ... Here's the full breakdown by dataset, since none of the articles make it clear -- Private Eval: - 85%: threshold for winning the prize [1] Semi-Private Eval: - 87.5%: o3 (unlimited compute) [2] - 75.7%: o3 (limited compute) [2] Public Eval: - 91.5%: o3 (unlimited compute) [2] - 82.8%: o3 (limited compute) [2] - 64.2%: human average (Mechanical Turk) [1] [3] Public…

Paper and discussion history. A mention is not an endorsement.

H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark — mentioned by hypoxia

Fri, 20 Dec 2024 20:24:29 GMT

It actually beats the human average by a wide margin: - 64.2% for humans vs. 82.8%+ for o3. ... Private Eval: - 85%: threshold for winning the prize [1] Semi-Private Eval: - 87.5%: o3 (unlimited compute) [2] - 75.7%: o3 (limited compute) [2] Public Eval: - 91.5%: o3 (unlimited compute) [2] - 82.8%: o3 (limited compute) [2] - 64.2%: human average (Mechanical Turk) [1] [3] Public Training: - 76.2%: human average (Mechanical Turk) [1] [3] ... References: [1] https://arcprize.org/guide [2] https://arcprize.org/blog/oai-o3-pub-breakthrough [3] https://arxiv.org/abs/2409.01374

Paper and discussion history. A mention is not an endorsement.

H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark — mentioned by obblekk

Fri, 20 Dec 2024 18:31:41 GMT

Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an eval of general reasoning ability, we have an algorithm for that now. There's a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI is within reach, now that we have an algorithm that works. Congrats to both Francois Chollet for developing this compelling eval, and to the researchers who saturated it! [1] https://x.com/SmokeAwayyy/status/1870171624403808366…

Paper and discussion history. A mention is not an endorsement.

H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark — mentioned by trott

Thu, 14 Nov 2024 01:47:16 GMT

> But if someone has already hit the human-level threshold There is some controversy over what the human-level threshold is. A recent and very extensive study measured just 60.2% using Amazon Mechanical Turkers, for the same setup [1]. But the Turkers had no prior experience with the dataset, and were only given 5 tasks each. Regardless, I believe ARC-AGI should aim for a higher threshold than what average humans achieve, because the ultimate goal of AGI is to supplement or replace high-IQ experts (who tend to do very well on ARC) --- [1] Table 1 in https://arxiv.org/abs/2409.01374 2-shot…

Paper and discussion history. A mention is not an endorsement.