Emergent Analogical Reasoning in Large Language Models · HackerPapers discussions

Emergent Analogical Reasoning in Large Language Models — mentioned by canjobear

Sun, 03 Mar 2024 17:28:58 GMT

On the other hand they’re about as good as humans on a text version of Raven’s matrices, another IQ test. https://arxiv.org/pdf/2212.09196v1.pdf

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Fri, 15 Dec 2023 02:30:05 GMT

>Every human is not even capable of reading, or speaking, or thinking at all. But those humans who can't are not the benchmark of general human intelligence. It's simple. If you're going to disqualify a potential general intelligence because of a particular task solving ability, it better be something every general intelligence can do. Now let's forget every comparison with the blind, disabled and so on. Let's compare with humans who according to you are "capable of reading, or speaking, or thinking at all ", what's the magic number? Do you really think every human capable of doing all three…

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Fri, 24 Nov 2023 21:38:57 GMT

Late reply but, LLMs are not bad at abstraction in general, just spatially focused abstraction. https://arxiv.org/abs/2212.09196 Also, how the data is presented matters a lot. Currently, LLMs handle the benchmark much better presented linearly in 1 dimension https://arxiv.org/abs/2305.18354

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Sun, 19 Nov 2023 18:00:25 GMT

>The problem with a statement like is that it leaves the door open to accepting any kind of canned generality as "abstraction in general". Not really https://arxiv.org/abs/2212.09196

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Sat, 02 Sep 2023 19:50:38 GMT

>They are good at writing code because that's very similar, but fall apart in things that need some actual abstract thinking, like math. Pretty odd assertion. LLMs are not "good at speech, bad at abstract thinking". What do these have to do with speech ? https://general-pattern-machines.github.io/ https://arxiv.org/abs/2212.09196 It doesn't even hold with your example because GPT-4 is pretty good at Math, nowhere near "falling apart".

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Mon, 22 May 2023 23:25:33 GMT

I don't think you took more than a passing glance, if any at those papers. What you describe is impossible with these 3. https://arxiv.org/abs/2212.09196 - new evaluation set introduced with the paper. modelled after tests that previously only had visual equivalents. contamination literally impossible https://arxiv.org/abs/2204.02329 - effect of explanations on questions introduced with the paper. dataset concerns make no sense. https://arxiv.org/abs/2211.09066 - new prompting method introduced to improve algorithmic calculations. dataset concerns make no sense. The Casual paper is the only…

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Mon, 22 May 2023 22:23:07 GMT

>It's not better at reasoning. It's barely even capable of it You are wrong. and there's many papers to show otherwise. Algorithmic, Casual, Inference, Analogical LLMs reason just fine https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2305.00050 https://arxiv.org/abs/2204.02329 https://arxiv.org/abs/2211.09066

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Sun, 21 May 2023 02:46:45 GMT

No world model ? A world model is so obvious papers like these are more confirmation than surprise https://arxiv.org/abs/2305.11169 https://arxiv.org/abs/2210.13382 There a certain sentiment that AGI however you wish to define it won't infact be a "We'll know it when we see it" situation but rather a "AGI will arrive long before consensus reaches its AGI". LLMs have made me believe this will 100% be the case, either way. It's one thing to argue over things we can't evaluate even now but man the 100th "They can't reason!" every week is pretty funny when you can basically take your pick of…

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Tue, 09 May 2023 01:03:35 GMT

LLMs are not very bad at reasoning. People say this but struggle to demonstrate how. There are many types of reasoning. For some like abstract analogical reasoning, LLMs are top % performers. https://arxiv.org/abs/2212.09196 There's many papers at this point demonstrating pretty strong reasoning. From Casual Reasoning, and explanatory reasoning to algorithmic reasoning. https://arxiv.org/abs/2305.00050 https://arxiv.org/abs/2204.02329 https://arxiv.org/abs/2211.09066 The only kind of reasoning that's somewhat lacking is logical reasoning. And even then it's not awful, just lagging behind…

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Thu, 20 Apr 2023 23:37:41 GMT

>But arguably GPT-4 is already at that level There really is no arguably about it. There isn't a verifiable definition/criteria for general intelligence that GPT-4 fails that a significant chunk of the human population doesn't also fail. GPT-4 performs nearly all tasks it's given at at least the average human level, usually well above the average baseline. In some cases (nlp, analogical reasoning - https://arxiv.org/abs/2212.09196, and some others it's top %) You basically have to make up your own definition that can't be substantiated (don't worry, plenty seem to do this). It almost always…

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — HN submission

Tue, 04 Apr 2023 12:54:13 GMT

GPT-3 outperforms humans on Raven's matrices, shows strong analogical reasoning

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Tue, 04 Apr 2023 12:44:20 GMT

You're right that LLMs can solve abstract analogical reasoning problems https://arxiv.org/abs/2212.09196 Another issue is the vision side. The vast majority of multimodal models are working on essentially an image to text objective task. That won't cut it here. We need the equivalent of an artificial visual cortex. We don't have that yet

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Tue, 04 Apr 2023 12:30:23 GMT

Yes. However, The vision problems will require something much more than an image to text objective task. It will require the equivalent of an artificial visual cortex. We don't have that yet. For abstract analogical reasoning, LLMs don't have a problem with that. https://arxiv.org/abs/2212.09196

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Tue, 04 Apr 2023 12:26:58 GMT

It is. Although I'd be careful with the conclusions. The problems are presented in a way that make it difficult to solve. The vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here. For the text there could be tokenizer issues. LLMs don't really have any problem with abstract analogical reasoning https://arxiv.org/abs/2212.09196

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Tue, 04 Apr 2023 12:15:14 GMT

That's not stuff an LLM can't do. It's just presented in a way that makes it difficult to do so. First the vision problems will require the equivalent of an artificial visual cortex, something we are seriously lacking in artificial intelligence at the moment. Image to text won't cut it here. For the text, LLMs don't really have any problem with analogical reasoning https://arxiv.org/abs/2212.09196

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Tue, 04 Apr 2023 04:52:44 GMT

>LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are statistical models, nothing more. "LLMs don't reason because they don't understand" is not the bastion of genius you think it is. It's a circular argument that relies on whatever bespoke interpretations you have cooked up. They don't form judgement or conclusions? Sure looks like they do. So what's the difference ? What is GPT-4 doing then when it correctly looks like it is reasoning and what's the difference between that and…

Paper and discussion history. A mention is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — HN submission

Mon, 27 Mar 2023 20:20:25 GMT

Emergent Analogical Reasoning in Large Language Models

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Emergent Analogical Reasoning in Large Language Models — mentioned by famouswaffles

Mon, 27 Mar 2023 00:03:33 GMT

Oh boy, if you want to fling papers/research, i can do that just fine. https://arxiv.org/abs/2212.09196v1 https://arxiv.org/abs/2211.09066 https://thegradient.pub/othello/ Moreover, "have trouble reasoning" and "can't reason at all" are two very different statements. Not that i particularly agree with either but you should keep that in mind.

Paper and discussion history. A mention is not an endorsement.