Evaluating Large Language Models in Theory of Mind Tasks · HackerPapers discussions

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by famouswaffles

Mon, 20 Nov 2023 23:26:23 GMT

>So when LLMs deceive, are they even aware of the effect the deception might have on the audience? Do they have any notion of manipulating the mental models of the audience? So..theory of mind ? https://arxiv.org/abs/2302.02083 https://arxiv.org/abs/2309.01660

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by netfortius

Sun, 14 May 2023 06:43:29 GMT

One of the articles linked here [1] reminded me of the great Ranulph Glanville, and his "Inside Every White Box There Are Two Black Boxes Trying To Get Out" [1] https://arxiv.org/abs/2302.02083

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by fl7305

Tue, 04 Apr 2023 06:46:15 GMT

> figuring out what people are thinking in person Have you seen the latest research, where GPT-4 is shown to have "Theory of Mind"-like ability (what people are thinking) at a human 7-year old level? https://arxiv.org/abs/2302.02083

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by slowmovintarget

Wed, 29 Mar 2023 15:42:54 GMT

Another poster shared a link to this paper last week for Theory of Mind: https://arxiv.org/abs/2302.02083 We're seeing those other capabilities emerge; like being able to play chess though it's not been trained to do so. That is, these LLMs are displaying emergent abilities associated with reasoning. These LLMs aren't R. Daneel Olivaw or R2D2 (which is what I think of when I think of the original term for AI, and what we took to calling AGI). We're closer to seeing the just-the-facts AIs we encounter in Blindsight. Intelligence without awareness. Funny that we still have to use science…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by stevenhuang

Thu, 23 Mar 2023 23:25:27 GMT

Welll.... Theory of Mind May Have Spontaneously Emerged in Large Language Models - https://arxiv.org/abs/2302.02083 Previously discussed - https://news.ycombinator.com/item?id=34730365

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by haskellandchill

Fri, 24 Feb 2023 18:58:13 GMT

We do have a way to verify understanding: causality. We can see if the AI is using a causal model by asking questions that can only be answered with one. Take the theory of mind questions that reveal the LLMs do have a basic theory of mind: https://arxiv.org/abs/2302.02083.

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by isaacfrond

Mon, 20 Feb 2023 15:58:34 GMT

Paper is here: https://arxiv.org/abs/2302.02083

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by zadler

Sun, 19 Feb 2023 06:44:51 GMT

Where do you draw the line on this “it’s just regurgitating” argument? https://arxiv.org/abs/2302.02083

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by stevenhuang

Sun, 19 Feb 2023 02:17:03 GMT

For all we know about intelligence/consciousness, it may be an emergent behaviour of complex systems. See this cool this paper, "Theory of Mind May Have Spontaneously Emerged in Large Language Models" https://arxiv.org/abs/2302.02083

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by agravier

Wed, 15 Feb 2023 20:05:28 GMT

It matters to understand how things work in order to understand their behaviour and react properly. I've seen people draw conclusions from applying Theory of Mind tests to LLMs (https://arxiv.org/abs/2302.02083). Those psychological were designed to assess humans psychological abilities or deficiencies, they assume that the language used by the human respondent reflects their continued and deep understanding of others' state of mind. In LLMs, there is no understanding involved. Dismissing the Chinese Room argument is an ostrich strategy. You're refusing to consider its lack of understanding…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by ec109685

Tue, 14 Feb 2023 01:46:27 GMT

There's more than comprehension. It can do some amount of higher order reasoning: https://arxiv.org/abs/2302.02083 "Theory of mind (ToM), or the ability to impute unobservable mental states to others, is central to human social interactions, communication, empathy, self-consciousness, and morality. We administer classic false-belief tasks, widely used to test ToM in humans, to several language models, without any examples or pre-training. Our results show that models published before 2022 show virtually no ability to solve ToM tasks. Yet, the January 2022 version of GPT-3 (davinci-002) solved…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by ec109685

Tue, 14 Feb 2023 01:39:36 GMT

Exactly, there is more there. Here's an example where it gets theory of mind questions right: https://arxiv.org/abs/2302.02083

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by sdenton4

Sun, 12 Feb 2023 00:36:57 GMT

"Theory of Mind May Have Spontaneously Emerged in Large Language Models" https://arxiv.org/abs/2302.02083 'Theory of mind (ToM), or the ability to impute unobservable mental states to others, is central to human social interactions, communication, empathy, self-consciousness, and morality. We administer classic false-belief tasks, widely used to test ToM in humans, to several language models, without any examples or pre-training. Our results show that models published before 2022 show virtually no ability to solve ToM tasks. Yet, the January 2022 version of GPT-3 (davinci-002) solved 70% of…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — mentioned by jointpdf

Sat, 11 Feb 2023 19:14:53 GMT

I see a lot of confident assertions of this type (LLMs don’t actually understand anything, cannot be creative, cannot be conscious, etc.), but never any data to substantiate the claim. This recent paper suggests that recent LLMs may be acquiring theory of mind (or something analogous to it): https://arxiv.org/abs/2302.02083v1 Some excerpts: > We administer classic false-belief tasks, widely used to test ToM in humans, to several language models, without any examples or pre-training. Our results show that models published before 2022 show virtually no ability to solve ToM tasks. Yet, the…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models in Theory of Mind Tasks — HN submission

Thu, 09 Feb 2023 19:57:05 GMT

Theory of Mind May Have Spontaneously Emerged in Large Language Models

170 points; 309 discussion replies on HN. Paper and source history. A submission is not an endorsement.