HackerPapers

arXiv:2212.09196cs.AI

Emergent Analogical Reasoning in Large Language Models

Taylor Webb, Keith J. Holyoak, Hongjing Lu

Abstract

The recent advent of large language models has reinvigorated debate over whether human cognitive capacities might emerge in such generic models given sufficient training data. Of particular interest is the ability of these models to reason about novel problems zero-shot, without any direct training. In human cognition, this capacity is closely tied to an ability to reason by analogy. Here, we performed a direct comparison between human reasoners and a large language model (the text-davinci-003 variant of GPT-3) on a range of analogical tasks, including a non-visual matrix reasoning task based on the rule structure of Raven's Standard Progressive Matrices. We found that GPT-3 displayed a surprisingly strong capacity for abstract pattern induction, matching or even surpassing human capabilities in most settings; preliminary tests of GPT-4 indicated even better performance. Our results indicate that large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems.

arXiv comments
Published at Nature Human Behaviour (2023) https://doi.org/10.1038/s41562-023-01659-w
Journal reference
DOI
Subjects
cs.AIArtificial Intelligence cs.CLComputation and Language

On Hacker News

Linked by 2 identifiable commenters in 16 comments across 11 threads, from Mar 2023 to Mar 2024. Found in 2 HN submissions from Mar 2023 to Apr 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments11 threads

  1. Top AIs still fail IQ tests 1 comment

    • On the other hand they’re about as good as humans on a text version of Raven’s matrices, another IQ test. https://arxiv.org/pdf/2212.09196v1.pdf

      canjobearView on HNParent context

  2. Weak-to-Strong Generalization 1 comment

    • >Every human is not even capable of reading, or speaking, or thinking at all. But those humans who can't are not the benchmark of general human intelligence. It's simple. If you're going to disqualify a potential general intelligence because of a particular task solving ability…

      famouswafflesView on HNParent context

  3. Sutskever: OpenAI board doing its mission to build AGI that benefits all 1 comment

    • Late reply but, LLMs are not bad at abstraction in general, just spatially focused abstraction. https://arxiv.org/abs/2212.09196 Also, how the data is presented matters a lot. Currently, LLMs handle the benchmark much better presented linearly in 1 dimension…

      famouswafflesView on HNParent context

  4. Comparing humans, GPT-4, and GPT-4V on abstraction and reasoning tasks 1 comment

    • >The problem with a statement like is that it leaves the door open to accepting any kind of canned generality as "abstraction in general". Not really https://arxiv.org/abs/2212.09196

      famouswafflesView on HNParent context

  5. Recursively summarizing enables long-term dialogue memory in LLMs 1 comment

    • >They are good at writing code because that's very similar, but fall apart in things that need some actual abstract thinking, like math. Pretty odd assertion. LLMs are not "good at speech, bad at abstract thinking". What do these have to do with speech ?…

      famouswafflesView on HNParent context

  6. Governance of Superintelligence 2 comments

    • >It's not better at reasoning. It's barely even capable of it You are wrong. and there's many papers to show otherwise. Algorithmic, Casual, Inference, Analogical LLMs reason just fine https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2305.00050…

      famouswafflesView on HNParent context

    • I don't think you took more than a passing glance, if any at those papers. What you describe is impossible with these 3. https://arxiv.org/abs/2212.09196 - new evaluation set introduced with the paper. modelled after tests that previously only had visual equivalents.…

      famouswafflesView on HNParent context

  7. Rodney Brooks on GPT-4 1 comment

    • No world model ? A world model is so obvious papers like these are more confirmation than surprise https://arxiv.org/abs/2305.11169 https://arxiv.org/abs/2210.13382 There a certain sentiment that AGI however you wish to define it won't infact be a "We'll know it when we see it"…

      famouswafflesView on HNParent context

  8. LLMs are not greater than the sum of their parts: researchers 1 comment

    • LLMs are not very bad at reasoning. People say this but struggle to demonstrate how. There are many types of reasoning. For some like abstract analogical reasoning, LLMs are top % performers. https://arxiv.org/abs/2212.09196 There's many papers at this point demonstrating pretty…

      famouswafflesView on HNParent context

  9. Large language models: Scaling laws and emergent properties 1 comment

    • >But arguably GPT-4 is already at that level There really is no arguably about it. There isn't a verifiable definition/criteria for general intelligence that GPT-4 fails that a significant chunk of the human population doesn't also fail. GPT-4 performs nearly all tasks it's…

      famouswafflesView on HNParent context

  10. Eight things to know about large language models [pdf] 5 comments

    • >LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more. "LLMs don't reason because they don't understand" is not the bastion of…

      famouswafflesView on HNParent context

    • That's not stuff an LLM can't do. It's just presented in a way that makes it difficult to do so. First the vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't…

      famouswafflesView on HNParent context

    • It is. Although I'd be careful with the conclusions. The problems are presented in a way that make it difficult to solve. The vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment.…

      famouswafflesView on HNParent context

    • Yes. However, The vision problems will require something much more than an image to text objective task. It will require the equivalent of an artificial visual cortex. We don't have that yet. For abstract analogical reasoning, LLMs don't have a problem with that.…

      famouswafflesView on HNParent context

    • You're right that LLMs can solve abstract analogical reasoning problems https://arxiv.org/abs/2212.09196 Another issue is the vision side. The vast majority of multimodal models are working on essentially an image to text objective task. That won't cut it here. We need the…

      famouswafflesView on HNParent context

  11. Capabilities of GPT-4 on Medical Challenge Problems 1 comment

    • Oh boy, if you want to fling papers/research, i can do that just fine. https://arxiv.org/abs/2212.09196v1 https://arxiv.org/abs/2211.09066 https://thegradient.pub/othello/ Moreover, "have trouble reasoning" and "can't reason at all" are two very different statements. Not that i…

      famouswafflesView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.