GPQA: A Graduate-Level Google-Proof Q&A Benchmark · HackerPapers discussions

GPQA: A Graduate-Level Google-Proof Q&A Benchmark — mentioned by ricardobeat

Mon, 08 Dec 2025 02:09:59 GMT

One of the reported hallucinations in this work [1], starting with David Rein, says the other authors are entirely made up. They are indeed absent from the original cited paper [2], but a Google search shows some of the same names featured in citations from other papers [3] [4]. Most of the names in these wrong attributions are actual people though, not hallucinations. What is going on? Is this a case of AI-powered citation management creating some weird feedback loop? [1] https://app.gptzero.me/documents/54c8aa45-c97d-48fc-b9d0-d49... [2] https://arxiv.org/pdf/2311.12022 [3]…

Paper and discussion history. A mention is not an endorsement.

GPQA: A Graduate-Level Google-Proof Q&A Benchmark — mentioned by vonneumannstan

Fri, 28 Mar 2025 14:40:05 GMT

>I’m not sure how you can have played with LLMs so much and missed this. I hope you don’t trust what they say about recipes or how to handle legal problems or how to clean things or how to treat disease or any fact-checking whatsoever. This is like a GPT3.5 level criticism. o1-pro is probably better at pure fact retrieval than most PhDs in any given field. I challenge you to try it. In fact take the GPQA test yourself and see how you do then give the same questions to o1. https://arxiv.org/pdf/2311.12022

Paper and discussion history. A mention is not an endorsement.

GPQA: A Graduate-Level Google-Proof Q&A Benchmark — mentioned by apsec112

Wed, 24 Apr 2024 02:34:52 GMT

Those are really sweeping conclusions, considering the experiment is just a single iteration of a single prompt! FWIW, Claude Opus got this for me on the first try: "In the Gilligan's Island episode "Seer Gilligan" (season 3, episode 8), Gilligan gains the ability to read minds after being hit on the head with a coconut. At first, the castaways are excited about Gilligan's new power and try to use it to their advantage. However, his mind-reading abilities soon cause chaos and misunderstandings among the group. In the end, Gilligan gets hit on the head again and loses his mind-reading powers…

Paper and discussion history. A mention is not an endorsement.

GPQA: A Graduate-Level Google-Proof Q&A Benchmark — mentioned by nopinsight

Mon, 04 Mar 2024 17:26:47 GMT

“Claude 3 gets ~60% accuracy on GPQA. It's hard for me to understate how hard these questions are—literal PhDs (in different domains from the questions) [spending over 30 minutes] with access to the internet get 34%. PhDs in the same domain (also with internet access!) get 65% - 75% accuracy.” — David Rein, first author of the GPQA Benchmark. I added text in […] based on the benchmark paper’s abstract. https://twitter.com/idavidrein/status/1764675668175094169 GPQA: A Graduate-Level Google-Proof Q&A Benchmark https://arxiv.org/abs/2311.12022

Paper and discussion history. A mention is not an endorsement.