HackerPapers

arXiv:2311.09247cs.AI

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks

Melanie Mitchell, Alessandro B. Palmarini, Arseny Moskvichev

Abstract

We explore the abstract reasoning abilities of text-only and multimodal versions of GPT-4, using the ConceptARC benchmark [10], which is designed to evaluate robust understanding and reasoning with core-knowledge concepts. We extend the work of Moskvichev et al. [10] by evaluating GPT-4 on more detailed, one-shot prompting (rather than simple, zero-shot prompts) with text versions of ConceptARC tasks, and by evaluating GPT-4V, the multimodal version of GPT-4, on zero- and one-shot prompts using image versions of the simplest tasks. Our experimental results support the conclusion that neither version of GPT-4 has developed robust abstraction abilities at humanlike levels.

arXiv comments
Corrected Figure 3 (extra spaces were replaced by commas, which were lost in original formatting)
Journal reference
Proceedings of the LLM-CP Workshop, AAAI 2024
DOI
Subjects
cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 10 comments across 9 threads, from Nov 2023 to Jun 2025. Found in 2 HN submissions from Nov 2023 to Nov 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Seven replies to the viral Apple reasoning paper and why they fall short 1 comment

    • Your entire edit essentially walks back your earlier strong claims. None of your current points actually support your position. 1. No, it doesn't. That's a ridiculous claim. Are you seriously suggesting that statistics require reasoning? 2. If you map that language to tokens…

      lossoloView on HNParent context

  2. Markov chains are funnier than LLMs 2 comments

    • > "generalize to its dataset" is a contradiction Not "to" but over, example the same code written in one language over the other language. > if you think being able to generalize in ways similar to the whole of the internet does not give your meaningful abilities to reason, I'm…

      lossoloView on HNParent context

    • > You can get them to solve unseen problems just fine Prove that the problem wasn't seen by them in other form. > Specify a grammar in BNF notation and tell it to generate or parse sentences for you. You can produce a more than random enough grammar that it it can't have derived…

      lossoloView on HNParent context

  3. OpenAI and Elon Musk 1 comment

    • There is evidence that they actually can't reason: https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809

      lossoloView on HNParent context

  4. New theory suggests LLMs can understand text 1 comment

    • For every N papers suggesting that LLM can "understand" you have X papers that suggests they can't. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809

      lossoloView on HNParent context

  5. Chess-GPT's Internal World Model 1 comment

    • There is also a lot of evidence lately that they do not generalize. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654

      lossoloView on HNParent context

  6. Will scaling work? 1 comment

    • A few more interesting papers not mentioned in the article: "Faith and Fate: Limits of Transformers on Compositionality" https://arxiv.org/abs/2305.18654 "Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks": https://arxiv.org/abs/2311.09247 "Embers of…

      lossoloView on HNParent context

  7. Details emerge of surprise board coup that ousted CEO Sam Altman at OpenAI 1 comment

    • Wikipedia has broader knowledge, and yet no one calls it intelligent. I'm talking about reasoning capability, and GPT-4 is well below human on complex tasks, especially multi-step tasks, which is why "autonomous agents" like AutoGPT, BabyAGI, etc are not yet very useful. For…

      reissbakerView on HNParent context

  8. I disagree with Geoff Hinton regarding "glorified autocomplete" 1 comment

    • There are plenty of results supporting my assertion; but the tests must be carefully designed. Of course, LLMs are not databases that store exact answers - so it's not enough to ask it something that it hasn't seen, if it's seen something similar (as is likely the case with your…

      skepticATXView on HNParent context

  9. Sutskever: OpenAI board doing its mission to build AGI that benefits all 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.