HackerPapers

arXiv:2201.11903cs.CL

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou

Abstract

We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 41 identifiable commenters in 47 comments across 43 threads, from Apr 2022 to Sep 2026. Found in 3 HN submissions from Apr 2022 to Jan 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments43 threads

  1. GPT-6 Astra 1 comment

    • "Chain of Thoughts" is a term from the title of a 2022 research paper "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (https://arxiv.org/abs/2201.11903), well before ChatGPT and the subsequent marketing hype. If anything, it's the most correct way to use…

      minimaxirView on HNParent context

  2. Understanding is the new bottleneck 1 comment

    • Jason Wei's chain-of-thought prompting techniques literally revolutionized llms themselves. https://arxiv.org/abs/2201.11903

      jfyiView on HNParent context

  3. The session you cannot take with you 1 comment

    • Reasoning improves the output of the model. Originally it was a prompting method called chain of thought but later models were trained to do the reasoning steps without the human prompting for it. Being able to see the reasoning steps is just an accidental benefit.…

      jurgenburgenView on HNParent context

  4. I admire Fabrice Bellard. He is almost certainly a better overall programmer 1 comment

    • My biggest gripe with the discourse around AI, especially by programmers with hubris about Machine Learning, is the idea that LLMs can’t come up with “novel solutions”. They can, and they have. CoT[0] is how LLMs can output tokens in “reasoning space” to guide their “thinking”…

      isomorphic_duckView on HNParent context

  5. Using “underdrawings” for accurate text and numbers 1 comment

    • Isn’t this sort of just “chain of thought” (i.e. the seminal https://arxiv.org/abs/2201.11903 ) where the user is helping the model 1-shot or k-shot the solution instead of 0-shot? I’ve used a similar technique to great effect. I feel things are so new / moving so fast that it’s…

      choppafaceView on HNParent context

  6. Claws are now a new layer on top of LLM agents 1 comment

    • AI from first principles has not changed. Fundamentally it is: neural nets, transformers and RL. The most important paper in recent years is on CoT [https://arxiv.org/pdf/2201.11903] and I'm not even sure what comes close. And I think what's more important these days is knowing…

      password54321View on HNParent context

  7. Ortega hypothesis 1 comment

    • Yeah. One example of people mindlessly mass citing some random paper is this: Chain of thought (CoT) prompting was used in the past to greatly enhance the reasoning ability of LLMs. Usually this paper is cited when CoT is discussed: https://arxiv.org/abs/2201.11903 It has over…

      cubefoxView on HNParent context

  8. I used o3 to find a remote zeroday in the Linux SMB implementation 1 comment

    • I'm not much of an ML engineer but I can point you to the original chain of thought paper [0] and Anthropic's docs on how to enable their official thinking scratchpad [1]. [0] https://arxiv.org/pdf/2201.11903 [1] https://docs.anthropic.com/en/docs/build-with-claude/extende...

      KTibowView on HNParent context

  9. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL 1 comment

    • Chain of Thought was known since 2022 (https://arxiv.org/abs/2201.11903), we just were stuck in a world where we were dumping more data and compute at the training instead of looking at other improvements.

      KubuxuView on HNParent context

  10. Use Prolog to improve LLM's reasoning 2 comments

    • COT = Chain-of-Thought https://arxiv.org/abs/2201.11903

      burntcaramelView on HNParent context

    • Of course Prolog is no magic but I'm sure you know the argument in favor of having LLMs generate Prolog are based on the observation that prompting or otherwise making an LLM to perform chain-of-thought reasoning results in demonstratable experimental improvements, with [1] the…

      tannhaeuserView on HNParent context

  11. Update on Reflection-70B 1 comment

    • No. CoT has been around for several years (Jan 2022) https://arxiv.org/abs/2201.11903. And so has Reflection (March 2023) https://arxiv.org/abs/2303.11366. This approach that was taken by Reflection is nothing new.

      ipsum2View on HNParent context

  12. Canvas is a new way to write and code with ChatGPT 1 comment

    • That prompts aren't science means little. If anything it makes them more important because you can't systematically arrive at good ones. If one spends a lot of time building an application to achieve an actual goal they'll realize the prompts make a gigantic difference and it…

      danielmarkbruceView on HNParent context

  13. Training Language Models to Self-Correct via Reinforcement Learning 1 comment

    • > as part of/after the main training I take this to mean during weight updates, e.g. training. > "runtime train of thought" I take runtime here to mean inference, not during RL. What does runtime mean to you? Previous approaches [0] successfully used inference time chain of…

      myownpetardView on HNParent context

  14. Grok-2 Beta Release 1 comment

    • My understanding is this used to be the case[1] but isn't really true any longer due to things like the "star" method for model training[2]. Empirically it absolutely (circa GPT3) used to be the case that if you prompted with "Explain all your reasoning step by step and then…

      seanhunterView on HNParent context

  15. What We Learned from a Year of Building with LLMs 1 comment

    • Justification before decision still works out better in practice, though, because of chain of thought [1]. You'll tend to get more accurate and better-justified decisions. With decision before justification, you tend to have a greater risk of the output being a wrong decision…

      bunderbunderView on HNParent context

  16. (Unsuccessfully) Fine-tuning GPT to play "Connections" 1 comment

    • I've found that for tasks like this you need to ask the model to output its reasoning prior to outputting the "answer". See "chain of thought" e.g https://arxiv.org/abs/2201.11903

      jappgarView on HNParent context

  17. OpenAI Begins Tackling ChatGPT Data Leak Vulnerability 1 comment

    • The verbosity certainly seems to me to be a specific change they have made. My guess is by getting it to do more "thinking out loud" they are trying to get it to do the kind of reasoning it used to only do when you prompted it to give its chain of thought, because that generally…

      seanhunterView on HNParent context

  18. OpenAI researchers warned board of AI breakthrough ahead of CEO ouster 1 comment

    • I quite agree and so would Wittgenstein, who (as I understand it) argued that precise language is essential to thought and reasoning[1]. I think one of the key things here is often what we think of as reasoning boils down to taking a problem in the real world and building a…

      seanhunterView on HNParent context

  19. GPT-4 can't reason 2 comments

    • I didn't have time to read this, but it is a single author paper, the author is not affiliated with a research group, it is not peer reviewed, it was published on a preprint server that I have never heard of. LLMs can definitely perform some kinds of reasoning. For example GSM8K…

      ma2rtenView on HNParent context

    • Obviously not. For example using Chain of Thought will increase the model's performance[1] [1] https://arxiv.org/abs/2201.11903

      redox99View on HNParent context

  20. Langchain Is Pointless 1 comment

    • Chain of thought: https://arxiv.org/abs/2201.11903 ReAct: https://arxiv.org/abs/2210.03629 Reflexion: https://arxiv.org/abs/2303.11366 Tree of thoughts: https://arxiv.org/abs/2305.10601 Good video on "Tree of thoughts" which also reviews / puts it in the context of other…

      krychuView on HNParent context

  21. GPT-4’s secret 2 comments

    • They’re referring to a method of prompting the model that encourages it to think through something step by step rather than spit out the answer. I think this is the paper that really kicked off this technique: https://arxiv.org/abs/2201.11903

      ickelbawdView on HNParent context

    • It has a specific technical connotation but it does map exactly to what you think it is.[1] Basically it's a trick that recognizes that language transformers only perform computation to generate words so for complex tasks you can get better results by asking the model to explain…

      seanhunterView on HNParent context

  22. LLMs are not greater than the sum of their parts: researchers 1 comment

    • Just speculating, but this looks like precisely the sort of result one might expect from fine-tuning models on chain-of-thought-prompted interactions like those described by Wei et al. in 2022[1]. Alternatively, as Madaan et al. show in their 2022 paper[2], this may simply be…

      cosmojgView on HNParent context

  23. A guide to prompting AI, for what it is worth 1 comment

    • That is important, but asking in certain ways can make a big difference. E.g. https://twitter.com/mitchellh/status/1645562198935347205. And just putting “let’s think step by step” at the end can really help some of the larger GPT models. https://arxiv.org/abs/2201.11903

      sp332View on HNParent context

  24. Teaching ChatGPT to speak my son’s invented language 1 comment
  25. GPT-4 Is a Reasoning Engine 2 comments

    • Decent read: this. Better read: the original paper demonstrating that prompting chain-of-thought reasoning elicits superior results. TLDR, give examples of working out the problem step-by-step, encourage GPT to emulate, and it's capable of solving challenges it would completely…

      xianshouView on HNParent context

    • It's not impossible to see that LLMs can make finite step "reasoning" between input and output, as each block of the transformer can model probabilistic causality. Transformers are sometimes considered a fully connected graph neural network, which can be used for modeling causal…

      BiasRegularizerView on HNParent context

  26. $335,000 Pay for ‘AI Whisperer’ Jobs Appears in Red-Hot Market 1 comment

    • Like the sister WaPo article hyping up this job title, also involving Anthropic (https://archive.is/pvwlx), this is still a super small market and only red-hot only because of Anthropic's job posting. It's like watching a penny stock have a huge price spike when there's one…

      vsaretoView on HNParent context

  27. GPT-4 performs significantly worse on coding problems not in its training data 1 comment

    • > but could be fixed How? The problem is known for a while, for example this article [0] mentions it (as Chain of Thought reasoning). You could think that just having a scratchpad of tokens is enough - you can arguably plan, backtrack and rewrite there [1], right? But this…

      DzugaruView on HNParent context

  28. We gave GPT-3.5 tools to run, write, commit, and deploy code 1 comment

    • > the first CoT paper introduced few shot examples It doesn't actually. See https://arxiv.org/pdf/2201.11903.pdf Perhaps you are thinking of the GPT-3 paper ("Language Models are Few-Shot Learners") which has plenty of these examples.

      nlView on HNParent context

  29. Build full “product skills” and you'll probably be fine 1 comment

    • Sure. A few below but far from exhaustive: - https://arxiv.org/abs/1909.07528 - https://arxiv.org/abs/2212.10403 - https://arxiv.org/abs/2201.11903 - https://arxiv.org/abs/2210.13382 There are also literally hundreds of articles and tweet threads about it. Moreover, as I said…

      soraki_soladeadView on HNParent context

  30. Oceania Always at War with Eastasia: Generative AI and Knowledge Pollution 1 comment

    • Here you go, here's a bunch of papers that you have not read. If you had read them then you would know that the errors are predictable and therefore there are many measurable ways to make improvements. Toolformer: Language Models Can Teach Themselves to Use Tools…

      williamcottonView on HNParent context

  31. Keep your AI claims in check 1 comment

    • Chain-of-thought is not the same with direct prompting. Step by step is better. > Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be…

      visargaView on HNParent context

  32. Our Society Is Not Prepared for This Much AI Awesome 1 comment

    • They literally "do context", as they do in-context learning. Also, factuality can effectively be solved by chain-of-thought prompting[0] and the use of external tools[1], also see toolformerzero[2] for a simple example. [0]: https://arxiv.org/abs/2201.11903 [1]…

      danielblnView on HNParent context

  33. Bing thinks Mars has 2.5B people based on an AI generated blog wrong answer 1 comment
  34. ChatGPT scores 80% on SAT reading/writing with collective chain of thought 1 comment

    • Using chain of thought [slightly modified] example from https://arxiv.org/abs/2201.11903 (ref'd in OP) Prompt: Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2…

      eternalbanView on HNParent context

  35. A new chat feature has been released by You Search 1 comment

    • If you ask it to write out the steps ahead of time you'll often get a more accurate answer [1] Q: "How far is Boston from Boston? Write down your steps before you answer." A: "Boston is a city located in the state of Massachusetts in the United States. To determine the distance…

      tuxracerView on HNParent context

  36. AI language models are struggling to “get” math 1 comment

    • Btw, here's an example of how even a very simple zero-shot/prompt-engineering attempt to introduce a bit of system 2 reasoning into a language model can improve results. https://arxiv.org/abs/2201.11903

      _0ffhView on HNParent context

  37. Am I doing it wrong? (OpenAI test) 1 comment

    • Yes, you are doing it wrong, but also this is a hard challenge. Here is some advice from a recent thread (https://twitter.com/goodside/status/1547613735543402500): > Note the useful deception at the end of this prompt: We tell the model what follows are correct training…

      yababa_yView on HNParent context

  38. Google fires engineer who called its AI sentient 1 comment

    • This has the same flavor as the initial criticisms of AlphaGo. "It will never be able to play anything other than Go", cue AlphaZero. "It will never be able to do so without being told the rules", cue MuZero. "It will never be able to do so without obscene amounts of data", cue…

      cmorezView on HNParent context

  39. Teaching GPT-3 to reverse words 1 comment

    • You're right for GPT 3, but it's an example of chain of thought reasoning, which seems to be a new area of research [1] and might get integrated into newer versions: [1] https://arxiv.org/abs/2201.11903

      skybrianView on HNParent context

  40. DeepMind: A Generalist Agent 1 comment

    • System 2 could be close behind System 2 by building on System 1 but it isn't clear how long that will take. This is happening since a few months ago: Wei et al (2022). Chain of Thought Prompting Elicits Reasoning in Large Language Models. https://arxiv.org/abs/2201.11903

      DavidSJView on HNParent context

Showing 40 of 43 threads. Download the full data to see all.

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.