Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks · HackerPapers discussions

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — mentioned by lossolo

Sun, 15 Jun 2025 14:32:23 GMT

Your entire edit essentially walks back your earlier strong claims. None of your current points actually support your position. 1. No, it doesn't. That's a ridiculous claim. Are you seriously suggesting that statistics require reasoning? 2. If you map that language to tokens, it's obvious the model will follow that mapping. etc. Here are papers showing that these models can't reason: https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809 You're mistaking pattern matching and the…

Paper and discussion history. A mention is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — mentioned by lossolo

Mon, 19 Aug 2024 12:26:29 GMT

> You can get them to solve unseen problems just fine Prove that the problem wasn't seen by them in other form. > Specify a grammar in BNF notation and tell it to generate or parse sentences for you. You can produce a more than random enough grammar that it it can't have derived the parsing of it from past text, but necessarily reasons about BNF notation sufficiently well to be able to use it to deduce the grammar, and use that to parse subsequent sentences. You can have it analyse them and tag them according to the grammar to. And generate sentences. Oh, come on. It's like rewriting the same…

Paper and discussion history. A mention is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — mentioned by lossolo

Mon, 19 Aug 2024 12:17:28 GMT

> "generalize to its dataset" is a contradiction Not "to" but over, example the same code written in one language over the other language. > if you think being able to generalize in ways similar to the whole of the internet does not give your meaningful abilities to reason, I'm not sure what I can tell you If after reading papers below that show empirically that they can't reason, you will still think they can reason, then I don't know what I can tell you. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654…

Paper and discussion history. A mention is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — mentioned by lossolo

Thu, 07 Mar 2024 12:02:24 GMT

There is evidence that they actually can't reason: https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809

Paper and discussion history. A mention is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — mentioned by lossolo

Mon, 22 Jan 2024 18:41:00 GMT

For every N papers suggesting that LLM can "understand" you have X papers that suggests they can't. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809

Paper and discussion history. A mention is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — mentioned by lossolo

Sun, 07 Jan 2024 21:20:19 GMT

There is also a lot of evidence lately that they do not generalize. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654

Paper and discussion history. A mention is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — mentioned by lossolo

Wed, 27 Dec 2023 17:32:22 GMT

A few more interesting papers not mentioned in the article: "Faith and Fate: Limits of Transformers on Compositionality" https://arxiv.org/abs/2305.18654 "Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks": https://arxiv.org/abs/2311.09247 "Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve" https://arxiv.org/abs/2309.13638 "Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models" https://arxiv.org/abs/2311.00871

Paper and discussion history. A mention is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — mentioned by reissbaker

Sun, 19 Nov 2023 20:31:38 GMT

Wikipedia has broader knowledge, and yet no one calls it intelligent. I'm talking about reasoning capability, and GPT-4 is well below human on complex tasks, especially multi-step tasks, which is why "autonomous agents" like AutoGPT, BabyAGI, etc are not yet very useful. For example: https://arxiv.org/abs/2311.09247

Paper and discussion history. A mention is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — HN submission

Sun, 19 Nov 2023 11:36:34 GMT

Comparing humans, GPT-4, and GPT-4V on abstraction and reasoning tasks

217 points; 177 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — mentioned by skepticATX

Sat, 18 Nov 2023 17:57:23 GMT

There are plenty of results supporting my assertion; but the tests must be carefully designed. Of course, LLMs are not databases that store exact answers - so it's not enough to ask it something that it hasn't seen, if it's seen something similar (as is likely the case with your programming language). One benchmark that I track closely is ConceptARC, which aims to test generalization and abstraction capabilities. Here is a very recent result that uses the benchmark: https://arxiv.org/abs/2311.09247. Humans correctly solved 91% of the problems, GPT-4 solved 33%, and GPT-4V did much worse than…

Paper and discussion history. A mention is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — mentioned by recitedropper

Sat, 18 Nov 2023 05:19:42 GMT

https://arxiv.org/pdf/2311.09247.pdf

Paper and discussion history. A mention is not an endorsement.

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks — HN submission

Fri, 17 Nov 2023 19:03:55 GMT

Comparing Humans, GPT-4, and GPT-4V on Abstraction and Reasoning Tasks

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.