HackerPapers

arXiv:2310.20216cs.AI

Does GPT-4 pass the Turing test?

Cameron R. Jones, Benjamin K. Bergen

Abstract

We evaluated GPT-4 in a public online Turing test. The best-performing GPT-4 prompt passed in 49.7% of games, outperforming ELIZA (22%) and GPT-3.5 (20%), but falling short of the baseline set by human participants (66%). Participants' decisions were based mainly on linguistic style (35%) and socioemotional traits (27%), supporting the idea that intelligence, narrowly conceived, is not sufficient to pass the Turing test. Participant knowledge about LLMs and number of games played positively correlated with accuracy in detecting AI, suggesting learning and practice as possible strategies to mitigate deception. Despite known limitations as a test of intelligence, we argue that the Turing test continues to be relevant as an assessment of naturalistic communication and deception. AI models with the ability to masquerade as humans could have widespread societal consequences, and we analyse the effectiveness of different strategies and criteria for judging humanlikeness.

arXiv comments
28 pages, 21 figures
Journal reference
DOI
Subjects
cs.AIArtificial Intelligence cs.CLComputation and Language

On Hacker News

Linked by 5 identifiable commenters in 6 comments across 6 threads, from Nov 2023 to Jun 2024. Found in 2 HN submissions from Nov 2023 to Nov 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. AI Hype is completely out of control – especially since ChatGPT-4o [video] 1 comment

    • "Does X look like Y" is always a continum. In test environments, players are judged human with these rates: * Real humans 66% * GPT-4: 49.7% * ELIZA: 22% * GPT-3.5: 20% https://arxiv.org/pdf/2310.20216 (I'm rather surprised by ELIZA beating 3.5, as were the researchers).…

      ben_wView on HNParent context

  2. AI or Ain't: Eliza 1 comment

    • "GPT-4 achieved a success rate of 41 percent, second only to actual humans" also feels like a (much bigger) lie of omission looking at the original paper. GPT4's performance was in the range of 6% to 41%, Eliza's 27% score sat in the upper middle of that range, and considering…

      notahackerView on HNParent context

  3. Stuff we figured out about AI in 2023 1 comment
  4. 1960s chatbot ELIZA beat OpenAI's GPT-3.5 in a recent Turing test study 1 comment

    • GPT4 might not be great at the Turing test either, it seems (https://arxiv.org/abs/2310.20216 and that is the simple non-adverserial version of the test).

      RandomLensmanView on HNParent context

  5. OpenAI's board has fired Sam Altman 1 comment

    • not yet: https://arxiv.org/abs/2310.20216 that being said, it is highly intelligent, capable of reasoning as well as a human, and passes IQ tests like GMAT and GRE at levels like the 97th percentile. most people who talk about Chat GPT don't even realize that GPT 4 exists and is…

      aidamanView on HNParent context

  6. GraphCast: AI model for weather forecasting 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.