HackerPapers

arXiv:2302.02083cs.CL

Evaluating Large Language Models in Theory of Mind Tasks

Michal Kosinski

Abstract

Eleven Large Language Models (LLMs) were assessed using a custom-made battery of false-belief tasks, considered a gold standard in testing Theory of Mind (ToM) in humans. The battery included 640 prompts spread across 40 diverse tasks, each one including a false-belief scenario, three closely matched true-belief control scenarios, and the reversed versions of all four. To solve a single task, a model needed to correctly answer 16 prompts across all eight scenarios. Smaller and older models solved no tasks; GPT-3-davinci-003 (from November 2022) and ChatGPT-3.5-turbo (from March 2023) solved 20% of the tasks; ChatGPT-4 (from June 2023) solved 75% of the tasks, matching the performance of six-year-old children observed in past studies. We explore the potential interpretation of these findings, including the intriguing possibility that ToM, previously considered exclusive to humans, may have spontaneously emerged as a byproduct of LLMs' improving language skills.

arXiv comments
TRY RUNNING ToM EXPERIMENTS ON YOUR OWN: The code and tasks used in this study are available at Colab (https://colab.research.google.com/drive/1ZRtmw87CdA4xp24DNS_Ik_uA2ypaRnoU). Don't worry if you are not an expert coder, you should be able to run this code with no-to-minimum Python skills. Or copy-paste the tasks to ChatGPT's web interface. Proceedings of the National Academy of Sciences (PNAS) 2024
Journal reference
DOI
doi:10.1073/pnas.2405460121
Subjects
cs.CLComputation and Language cs.CYComputers and Society cs.HCHuman-Computer Interaction

On Hacker News

Linked by 12 identifiable commenters in 14 comments across 11 threads, from Feb 2023 to Nov 2023. Found in 1 HN submission from Feb 2023 to Feb 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments11 threads

  1. Misalignment and Deception by an autonomous stock trading LLM agent 1 comment

    • >So when LLMs deceive, are they even aware of the effect the deception might have on the audience? Do they have any notion of manipulating the mental models of the audience? So..theory of mind ? https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2309.01660

      famouswafflesView on HNParent context

  2. This time, it feels different 1 comment

    • One of the articles linked here [1] reminded me of the great Ranulph Glanville, and his "Inside Every White Box There Are Two Black Boxes Trying To Get Out" [1] https://arxiv.org/abs/2302.02083

      netfortiusView on HNParent context

  3. AI won't steal your job, people leveraging AI will 1 comment

    • > figuring out what people are thinking in person Have you seen the latest research, where GPT-4 is shown to have "Theory of Mind"-like ability (what people are thinking) at a human 7-year old level? https://arxiv.org/abs/2302.02083

      fl7305View on HNParent context

  4. Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models 1 comment

    • Another poster shared a link to this paper last week for Theory of Mind: https://arxiv.org/abs/2302.02083 We're seeing those other capabilities emerge; like being able to play chess though it's not been trained to do so. That is, these LLMs are displaying emergent abilities…

      slowmovintargetView on HNParent context

  5. Microsoft's paper on OpenAI's GPT-4 had hidden information 1 comment

    • Welll.... Theory of Mind May Have Spontaneously Emerged in Large Language Models - https://arxiv.org/abs/2302.02083 Previously discussed - https://news.ycombinator.com/item?id=34730365

      stevenhuangView on HNParent context

  6. Don't believe ChatGPT – we do not offer a "phone lookup" service 1 comment

    • We do have a way to verify understanding: causality. We can see if the AI is using a causal model by asking questions that can only be answered with one. Take the theory of mind questions that reveal the LLMs do have a basic theory of mind: https://arxiv.org/abs/2302.02083.

      haskellandchillView on HNParent context

  7. Artificial Intelligence Suddenly Evolves to Reach Theory of Mind 1 comment
  8. Researchers Discover a More Flexible Approach to Machine Learning 2 comments

    • For all we know about intelligence/consciousness, it may be an emergent behaviour of complex systems. See this cool this paper, "Theory of Mind May Have Spontaneously Emerged in Large Language Models" https://arxiv.org/abs/2302.02083

      stevenhuangView on HNParent context

    • Where do you draw the line on this “it’s just regurgitating” argument? https://arxiv.org/abs/2302.02083

      zadlerView on HNParent context

  9. Bing: “I will not harm you unless you harm me first” 1 comment

    • It matters to understand how things work in order to understand their behaviour and react properly. I've seen people draw conclusions from applying Theory of Mind tests to LLMs (https://arxiv.org/abs/2302.02083). Those psychological were designed to assess humans psychological…

      agravierView on HNParent context

  10. Bing AI can't be trusted 2 comments

    • Exactly, there is more there. Here's an example where it gets theory of mind questions right: https://arxiv.org/abs/2302.02083

      ec109685View on HNParent context

    • There's more than comprehension. It can do some amount of higher order reasoning: https://arxiv.org/abs/2302.02083 "Theory of mind (ToM), or the ability to impute unobservable mental states to others, is central to human social interactions, communication, empathy…

      ec109685View on HNParent context

  11. I had ChatGPT write a Decemberists song and recorded it 2 comments

    • I see a lot of confident assertions of this type (LLMs don’t actually understand anything, cannot be creative, cannot be conscious, etc.), but never any data to substantiate the claim. This recent paper suggests that recent LLMs may be acquiring theory of mind (or something…

      jointpdfView on HNParent context

    • "Theory of Mind May Have Spontaneously Emerged in Large Language Models" https://arxiv.org/abs/2302.02083 'Theory of mind (ToM), or the ability to impute unobservable mental states to others, is central to human social interactions, communication, empathy, self-consciousness…

      sdenton4View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.