HackerPapers

arXiv:2112.00114cs.LG

Show Your Work: Scratchpads for Intermediate Computation with Language Models

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, Augustus Odena

Abstract

Large pre-trained language models perform remarkably well on tasks that can be done "in one pass", such as generating realistic text or synthesizing computer programs. However, they struggle with tasks that require unbounded multi-step computation, such as adding integers or executing programs. Surprisingly, we find that these same models are able to perform complex multi-step computations -- even in the few-shot regime -- when asked to perform the operation "step by step", showing the results of intermediate computations. In particular, we train transformers to perform multi-step computations by asking them to emit intermediate computation steps into a "scratchpad". On a series of increasingly complex tasks ranging from long addition to the execution of arbitrary programs, we show that scratchpads dramatically improve the ability of language models to perform multi-step computations.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.NENeural and Evolutionary Computing

On Hacker News

Linked by 4 identifiable commenters in 5 comments across 5 threads, from Feb 2022 to Apr 2026. Found in 1 HN submission from Jul 2022 to Jul 2022. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Caveman: Why use many token when few token do trick 1 comment

    • Let’s see, I think these pretty much map out a little chronology of the research: https://arxiv.org/abs/2112.00114 https://arxiv.org/abs/2406.06467 https://arxiv.org/abs/2404.15758 https://arxiv.org/abs/2512.12777 First that scratchpads matter, then why they matter, then that…

      Chance-DeviceView on HNParent context

  2. Claude Opus 4.5 1 comment

    • Does anyone here understand "interleaved scratchpads" mentioned at the very bottom of the footnotes: > All evals were run with a 64K thinking budget, interleaved scratchpads, 200K context window, default effort (high), and default sampling settings (temperature, top_p). I…

      syncView on HNParent context

  3. ChatGPT has trouble giving an answer before explaining its reasoning 1 comment

    • This can certainly be done. Here's one example from 2021 demonstrating training an LLM to use a scratchpad ("talking to itself") to greatly improve accuracy on arithmetic problems: https://arxiv.org/pdf/2112.00114.pdf

      jncratonView on HNParent context

  4. I asked GPT-NeoX-20B a hundred arithmetic questions 1 comment

    • Yup, quite possible that this has something to do with it. There is other work showing that giving LMs a "scratchpad" for intermediate computations allows them to do much better not just at arithmetic but also things like predicting the output of some code…

      moyixView on HNParent context

  5. AlphaCode as a dog speaking mediocre English 1 comment

    • Sure, I think this would be a really interesting study to do! I have been meaning to scrape a big chunk of GitHub in order to do this kind of analysis. I think I would be surprised (just based on my own use of Codex and Copilot) if they were copying at the level of "same code…

      moyixView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.