HackerPapers

arXiv:2309.13638cs.CL

Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve

R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, Thomas L. Griffiths

Abstract

The widespread adoption of large language models (LLMs) makes it important to recognize their strengths and limitations. We argue that in order to develop a holistic understanding of these systems we need to consider the problem that they were trained to solve: next-word prediction over Internet text. By recognizing the pressures that this task exerts we can make predictions about the strategies that LLMs will adopt, allowing us to reason about when they will succeed or fail. This approach - which we call the teleological approach - leads us to identify three factors that we hypothesize will influence LLM accuracy: the probability of the task to be performed, the probability of the target output, and the probability of the provided input. We predict that LLMs will achieve higher accuracy when these probabilities are high than when they are low - even in deterministic settings where probability should not matter. To test our predictions, we evaluate two LLMs (GPT-3.5 and GPT-4) on eleven tasks, and we find robust evidence that LLMs are influenced by probability in the ways that we have hypothesized. In many cases, the experiments reveal surprising failure modes. For instance, GPT-4's accuracy at decoding a simple cipher is 51% when the output is a high-probability word sequence but only 13% when it is low-probability. These results show that AI practitioners should be careful about using LLMs in low-probability situations. More broadly, we conclude that we should not evaluate LLMs as if they are humans but should instead treat them as a distinct type of system - one that has been shaped by its own particular set of pressures.

arXiv comments
50 pages plus 11 page of references and 23 pages of appendices
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 2 identifiable commenters in 8 comments across 7 threads, from Dec 2023 to Jun 2025. Found in 2 HN submissions from Oct 2023 to Oct 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Seven replies to the viral Apple reasoning paper and why they fall short 1 comment

    • Your entire edit essentially walks back your earlier strong claims. None of your current points actually support your position. 1. No, it doesn't. That's a ridiculous claim. Are you seriously suggesting that statistics require reasoning? 2. If you map that language to tokens…

      lossoloView on HNParent context

  2. Markov chains are funnier than LLMs 2 comments

    • > "generalize to its dataset" is a contradiction Not "to" but over, example the same code written in one language over the other language. > if you think being able to generalize in ways similar to the whole of the internet does not give your meaningful abilities to reason, I'm…

      lossoloView on HNParent context

    • > You can get them to solve unseen problems just fine Prove that the problem wasn't seen by them in other form. > Specify a grammar in BNF notation and tell it to generate or parse sentences for you. You can produce a more than random enough grammar that it it can't have derived…

      lossoloView on HNParent context

  3. Sam Altman's under pressure amid questions about OpenAI's commitment to safety 1 comment

    • Yes, specifically it seemed like OpenAI was actively encouraging people (men) to have fake personal relationships with a chatbot. I am wondering if Sam Altman gave up on the idea that transformers can ever be general-purpose problem solvers[1] and is pivoting to the creepy…

      nicklecompteView on HNParent context

  4. OpenAI and Elon Musk 1 comment

    • There is evidence that they actually can't reason: https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809

      lossoloView on HNParent context

  5. New theory suggests LLMs can understand text 1 comment

    • For every N papers suggesting that LLM can "understand" you have X papers that suggests they can't. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809

      lossoloView on HNParent context

  6. Chess-GPT's Internal World Model 1 comment

    • There is also a lot of evidence lately that they do not generalize. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654

      lossoloView on HNParent context

  7. Will scaling work? 1 comment

    • A few more interesting papers not mentioned in the article: "Faith and Fate: Limits of Transformers on Compositionality" https://arxiv.org/abs/2305.18654 "Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks": https://arxiv.org/abs/2311.09247 "Embers of…

      lossoloView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.