GAIA: a benchmark for General AI Assistants · HackerPapers discussions

GAIA: a benchmark for General AI Assistants — mentioned by d4rkp4ttern

Sun, 16 Feb 2025 12:42:19 GMT

I’ve seen at least one deep-research replicator claiming they were the “best open deep research” tool on the GAIA benchmark: https://huggingface.co/papers/2311.12983 This is not a perfect benchmark but the closest I’ve seen.

Paper and discussion history. A mention is not an endorsement.

GAIA: a benchmark for General AI Assistants — HN submission

Fri, 24 Nov 2023 09:17:33 GMT

Gaia: A Benchmark for General AI Assistants

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

GAIA: a benchmark for General AI Assistants — mentioned by doctoboggan

Thu, 23 Nov 2023 06:15:45 GMT

Here are some example questions from the paper[0] Level 1 Question: What was the actual enrollment count of the clinical trial on H. pylori in acne vulgaris patients from Jan-May 2018 as listed on the NIH website? Ground truth: 90 Level 2 <photo of ice cream container showing nutrition facts> Question: If this whole pint is made up of ice cream, how many percent above or below the US federal standards for butterfat content is it when using the standards as reported by Wikipedia in 2020? Answer as + or - a number rounded to one decimal place. Ground truth: +4.6 Level 3 Question: In NASA’s…

Paper and discussion history. A mention is not an endorsement.

GAIA: a benchmark for General AI Assistants — HN submission

Thu, 23 Nov 2023 03:43:15 GMT

Meta: Gaia - A Benchmark for General AI Assistants

36 points; 8 discussion replies on HN. Paper and source history. A submission is not an endorsement.