HackerPapers

arXiv:2305.18654cs.CL

Faith and Fate: Limits of Transformers on Compositionality

Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, Yejin Choi

Abstract

Transformer large language models (LLMs) have sparked admiration for their exceptional performance on tasks that demand intricate multi-step reasoning. Yet, these models simultaneously show failures on surprisingly trivial problems. This begs the question: Are these errors incidental, or do they signal more substantial limitations? In an attempt to demystify transformer LLMs, we investigate the limits of these models across three representative compositional tasks -- multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer. We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures. Our empirical findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills. To round off our empirical study, we provide theoretical arguments on abstract multi-step reasoning problems that highlight how autoregressive generations' performance can rapidly decay with\,increased\,task\,complexity.

arXiv comments
10 pages + appendix (40 pages)
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 8 identifiable commenters in 14 comments across 13 threads, from Jun 2023 to Oct 2025. Found in 4 HN submissions from Jun 2023 to Apr 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments13 threads

  1. OpenAI researcher announced GPT-5 math breakthrough that never happened 1 comment

    • > LLMs can't do math. Ignoring conversations about 'reasoning', at a fundamental level LLMs do not 'do math' in the way that a calculator or a human does math. Sure we can train bigger and bigger models that give you the impression of this but there are proofs out there that…

      travelalbertaView on HNParent context

  2. Seven replies to the viral Apple reasoning paper and why they fall short 1 comment

    • Your entire edit essentially walks back your earlier strong claims. None of your current points actually support your position. 1. No, it doesn't. That's a ridiculous claim. Are you seriously suggesting that statistics require reasoning? 2. If you map that language to tokens…

      lossoloView on HNParent context

  3. Recent results show that LLMs struggle with compositional tasks 1 comment

    • That’s a different paper than the one this article describes. The article describes this paper: https://arxiv.org/abs/2305.18654

      janalsncmView on HNParent context

  4. Transformer^2: Self-Adaptive LLMs 1 comment

    • Would love to know if you know any other papers like: Faith and Fate: Limits of Transformers on Compositionality https://arxiv.org/abs/2305.18654 Maybe the analogy is something with gold mining. We could pretend that the machines that mine gold are actually creating gold.…

      monophonicaView on HNParent context

  5. Markov chains are funnier than LLMs 2 comments

    • > "generalize to its dataset" is a contradiction Not "to" but over, example the same code written in one language over the other language. > if you think being able to generalize in ways similar to the whole of the internet does not give your meaningful abilities to reason, I'm…

      lossoloView on HNParent context

    • > You can get them to solve unseen problems just fine Prove that the problem wasn't seen by them in other form. > Specify a grammar in BNF notation and tell it to generate or parse sentences for you. You can produce a more than random enough grammar that it it can't have derived…

      lossoloView on HNParent context

  6. Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking 1 comment

    • This paper does indeed follow your intuition to investigate the limits of transformers on compositional tasks (i.e., those that require multi-step reasoning, including your multiplication example): https://arxiv.org/abs/2305.18654 > Our empirical findings suggest that…

      blackbear_View on HNParent context

  7. OpenAI and Elon Musk 1 comment

    • There is evidence that they actually can't reason: https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809

      lossoloView on HNParent context

  8. New theory suggests LLMs can understand text 1 comment

    • For every N papers suggesting that LLM can "understand" you have X papers that suggests they can't. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809

      lossoloView on HNParent context

  9. Chess-GPT's Internal World Model 1 comment

    • There is also a lot of evidence lately that they do not generalize. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654

      lossoloView on HNParent context

  10. Will scaling work? 1 comment

    • A few more interesting papers not mentioned in the article: "Faith and Fate: Limits of Transformers on Compositionality" https://arxiv.org/abs/2305.18654 "Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks": https://arxiv.org/abs/2311.09247 "Embers of…

      lossoloView on HNParent context

  11. Copy is all you need 1 comment

    • What about LLM reasoning ability? Faith and Fate: Limits of Transformers on Compositionality https://arxiv.org/abs/2305.18654 Transformers solve compositional reasoning tasks by reducing multi-step compositional reasoning into linearized subgraph matching without problem-solving…

      MAXPOOLView on HNParent context

  12. Researchers warn of ‘model collapse’ as AI trains on AI-generated content 1 comment

    • The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. One way to think of why this is the case is because transformers are learning shortcuts to multistep reasoning tasks [1, 2], but do not learn…

      bmc7505View on HNParent context

  13. Weird GPT-4 behavior for the specific string “ davidjl” 1 comment

    • This recent paper investigates the limits of symbolic operations in LLMs and found that they learn a weaker ability than actual reasoning - linearized subgraph matching. > "Faith and Fate: Limits of Transformers on Compositionality" - We investigate the limits of these models…

      visargaView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.