Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models · HackerPapers discussions

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models — mentioned by bryan0

Fri, 25 Apr 2025 15:30:06 GMT

Here is the paper written about this prompt: https://arxiv.org/html/2406.02061v1

Paper and discussion history. A mention is not an endorsement.

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models — mentioned by warkdarrior

Wed, 04 Dec 2024 17:37:15 GMT

Did you link the wrong Arxiv paper? https://arxiv.org/abs/2406.02061 does not look like a position paper nor does it share any authors with this Genie 2 work.

Paper and discussion history. A mention is not an endorsement.

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models — mentioned by artninja1988

Wed, 04 Dec 2024 16:04:09 GMT

Looking at the list of authors, is this from their open endedness team? I found their position paper on it super convincing https://arxiv.org/abs/2406.02061

Paper and discussion history. A mention is not an endorsement.

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models — mentioned by s-macke

Fri, 11 Oct 2024 13:14:03 GMT

These results are very similar to the "Alice in Wonderland" problem [1, 2], which was already discussed a few months ago. However the authors of the other paper are much more critical and call it a "Complete Reasoning Breakdown". You could argue that the issue lies in the models being in an intermediate state between pattern matching and reasoning. To me, such results indicate that you can't trust any LLM benchmark results related to math and reasoning when you see, that changing the characters, numbers or the sentence structure in a problem alter the outcome by more than 20 percentage…

Paper and discussion history. A mention is not an endorsement.

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models — mentioned by smusamashah

Thu, 05 Sep 2024 22:00:50 GMT

We need results from these harder/different benchmarks which give pretty bad scores to current top LLMs. https://www.wolfram.com/llm-benchmarking-project/ https://help.kagi.com/kagi/ai/llm-benchmark.html Edit : There are few other benchmarks that give pretty low scores (<20%) to top LLMs. Can't find them atm. There was a benchmark with common sense easy looking questions. Edit: found two more papers https://arxiv.org/html/2405.19616 https://arxiv.org/html/2406.02061v1 Edit: How about Wordle? https://www.strangeloopcanon.com/p/what-can-llms-never-do https://news.ycombinator.com/item?id=40179232

Paper and discussion history. A mention is not an endorsement.

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models — mentioned by discreteevent

Mon, 08 Jul 2024 15:58:21 GMT

Well we don't have an understanding of how the brain works so we can't be fully sure but it's clear why they have this intuition: 1) Many people have had to cram for some exam where they didn't have time to fully understand the material. So for those parts they memorized as much as they could and got through the exam by pattern matching. But they knew there was a difference because they knew what it was like to fully understand something where that they could reason about it and play with it in their mind. 2) Crucially, if they understand the key mechanism early then they often don't need to…

Paper and discussion history. A mention is not an endorsement.

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models — mentioned by aadityaubhat

Thu, 27 Jun 2024 15:04:47 GMT

https://arxiv.org/abs/2406.02061v1 research paper shows the reasoning breakdown in SOTA LLMs by asking a simple question, “Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have?” I investigated performance of different prompts on this question, and show that 'Expand-then solve' prompt significantly outperforms standard and chain-of-thought prompts.

Paper and discussion history. A mention is not an endorsement.

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models — mentioned by salamo

Sun, 09 Jun 2024 02:26:01 GMT

What transformers have taught us is that gobbling up the whole internet can get you pretty far, but it doesn't change the fundamentals. Hallucinations still are trivial [1] to elicit. On the other hand, transformers are a software breakthrough that made training on the whole internet even feasible. Other methods (RNNs and LSTMs) were simply too slow. Transformers are a new hammer for an internet-sized nail. If I can make an analogy to chess engines, a type of AI so old that people don't call it AI anymore [2], if we had started out by applying transformers to available chess games, we would…

Paper and discussion history. A mention is not an endorsement.

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models — HN submission

Wed, 05 Jun 2024 14:13:39 GMT

Simple tasks showing reasoning breakdown in state-of-the-art LLMs

375 points; 380 discussion replies on HN. Paper and source history. A submission is not an endorsement.