GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models · HackerPapers discussions

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by delis-thumbs-7e

Mon, 06 Jul 2026 22:38:47 GMT

I am no linguist, but I believe this is referred to as surface structure and deep structure. What you describe is a line of text that is grammatically somewhat adequate to pass as readable and you treat all text the same. when we read the text, we decipher meaning out of according to word references and syntax - just like with Python or C++. However, if this were solely the case, we could not read Finnegan’s Wake. Probably vast majority of modern poetry would be unread by anyone, as would be pretty much all major works of philosophy Kant onwards. Deep structure is what according to Chomsky et…

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by _petronius

Mon, 09 Jun 2025 11:39:38 GMT

> The article skirts around a central question: what defines humans? Specifically, intelligence and emotions? > The entire article is saying "it looks kinds like a human in some ways, but people are being fooled!" > You can't really say that without at least attempting the admittedly very deep question of what an authentic human is. > To me, it's intelligent because I can't distinguish its output from a person's output, for much of the time. I think the article does address that rather directly, and that it is also is addressing very specifically your setence about what you can and can't…

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by MontyCarloHall

Sat, 07 Jun 2025 12:23:09 GMT

A slightly less cynical take is that they want to temper expectations for the capabilities of LLMs in people’s day-to-day lives, specifically in the context of Apple products. A “smarter Siri” is never going to be an autonomous personal assistant à la Jarvis from Iron Man, which seems to be where a lot of investors think things are going. That tracks with this [0] preprint also released by Apple a few months ago. A slightly more cynical take is that you’re absolutely correct, and making excuses for weak machine learning prowess has long been an Apple tenet. Recall that Apple never made…

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by efitz

Fri, 07 Feb 2025 18:39:38 GMT

What everyone needs to understand about [reasoning] LLMs is that LLMs can’t reason. https://arxiv.org/pdf/2410.05229

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by Clubber

Thu, 02 Jan 2025 02:19:28 GMT

>we don't know that they're capable of reasoning Apple AI researchers released a paper on it. They say no. https://arxiv.org/pdf/2410.05229

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by bwfan123

Wed, 01 Jan 2025 18:41:17 GMT

this work is similar to the GSM symbolic paper (applied to putnam) https://arxiv.org/html/2410.05229v1 going forward, llm performance must be reported on the confounded benchmark as well

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by intelkishan

Wed, 01 Jan 2025 13:05:35 GMT

I think you are probably referring to the following paper: https://arxiv.org/abs/2410.05229

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by coldtea

Mon, 23 Dec 2024 11:06:07 GMT

It's not like we discovered hot air ballons, and some people think we'll get to Moon and Mars with them... > Does person 1 have the burden of proof that rockets can be used to put things in orbit? Sure, but that doesn't make the reasoning used by person 2 valid to explain why person 1 is wrong. The reasoning by person 2 doesn't matter as much if 1 is making an ubsubstantiated claim to begin with. >There is no law that saws that intelligence requires anything but a large amount of data and computation. If you can show why these things are not sufficient, I am eager to read about it. Errors…

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by loxias

Sat, 21 Dec 2024 00:55:50 GMT

NovelQA is a great one! I also like GSM-Symbolic -- a benchmark based on making _symbolic templates_ of quite easy questions, and sampling them repeatedly, varying things like which proper nouns are used, what order relevant details appear, how many irrelevant details (GSM-NoOp) and where they are in the question, things like that. LLMs are far, _far_ below human on elementary problems, once you allow any variation and stop spoonfeeding perfectly phrased word problems. :) https://machinelearning.apple.com/research/gsm-symbolic https://arxiv.org/pdf/2410.05229 Paper came out in October, I…

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by zamalek

Mon, 02 Dec 2024 15:09:10 GMT

https://arxiv.org/abs/2410.05229 An 18% drop in accuracy (figure 8) is not insignificant. Even 4o suffered 10% loss (figure 6), and 4o isn't a small llm. Competent performance should have near zero performance loss. The simplest benchmark merely changes things like "john had 4 apples" to "Mary had 4 oranges." Performance loss due to inconsequential tokens changing is the very definition of over-fitting.

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by griomnib

Thu, 21 Nov 2024 20:04:08 GMT

Mathematical reasoning is the most obvious area where it breaks down. This paper does an excellent job of proving this point with some elegant examples: https://arxiv.org/pdf/2410.05229

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by verteu

Thu, 14 Nov 2024 22:45:23 GMT

(for reference: https://arxiv.org/pdf/2410.05229 )

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by ninjin

Thu, 07 Nov 2024 08:08:37 GMT

Thanks, that is probably the one. Here is a link to the paper itself on arXiv: https://arxiv.org/abs/2410.05229

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by turtlesdown11

Wed, 30 Oct 2024 18:15:04 GMT

did you not read https://arxiv.org/pdf/2410.05229 "We hypothesize that this decline is due to the fact that current LLMs are not capable of genuine logical reasoning; instead, they attempt to replicate the reasoning steps observed in their training data."

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by carschno

Wed, 23 Oct 2024 07:51:25 GMT

Indeed, test data like this constantly leaks into the training data, so these leaderboards are not necessarily representative for real-world problems. A better approach is to use variable evaluation like GSM-Symbolic (for evaluating mathematic reasoning): https://arxiv.org/abs/2410.05229

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by efitz

Mon, 21 Oct 2024 17:34:50 GMT

These are interesting tests but my question is, why are we doing them? LLM models are not “intelligent” by any meaningful measurement- they are not sapient/sentient/conscious/self-aware. They have no “intent” other than what was introduced to them via the system prompt. They cannot reason [1]. Are researchers worried about sapience/consciousness as an emergent property? Humans who are not AI researchers generally do not have good intuition or judgment about what these systems can do and how they will “fail” (perform other than as intended). However the cat is out of the bag already and it’s…

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by mannykannot

Sun, 20 Oct 2024 15:52:53 GMT

I feel you are putting too much emphasis on the importance and primacy of having a definition of words like 'reasoning'. As humanity has struggled to understand the world, it has frequently given names to concepts that seem to matter, well before it is capable of explaining with any sort of precision what these things are, and what makes them matter - take the word 'energy', for example. It seems clear to me that one must have these vague concepts before one can begin to to understand them, and also that it would be bizarre not to give them a name at that point - and so, at that point, we…

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by airstrike

Sat, 19 Oct 2024 16:23:06 GMT

Apple's study suggested a 0.3% drop in benchmark results for GPT-4o from fuzzying the prompts, which to me really means no drop and actually shows the benchmark results are more robust than I'd previously expect. It also only looked at GPT-4o, ignoring the allegedly more expensive GPT-4. Finally, I can't help but notice the irony of an article about LLMs that likes to "delve" into "crucial" points "to ensure"... and repeatedly defines the "large language models (LLMs)" acronym from the very first paragraph to the conclusion. It also claims the study was led by Mehrdad Farajtabar when he's…

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — mentioned by shakna

Tue, 15 Oct 2024 06:32:57 GMT

Pretty much everything shown here [0] [0] https://arxiv.org/pdf/2410.05229

Paper and discussion history. A mention is not an endorsement.

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models — HN submission

Fri, 11 Oct 2024 11:55:06 GMT

Understanding the Limitations of Mathematical Reasoning in LLMs

282 points; 266 discussion replies on HN. Paper and source history. A submission is not an endorsement.