HackerPapers

arXiv:2206.14858cs.CL

Solving Quantitative Reasoning Problems with Language Models

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, Vedant Misra

Abstract

Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.

arXiv comments
12 pages, 5 figures + references and appendices
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 5 comments across 5 threads, from Oct 2022 to Jan 2025. Found in 1 HN submission from Sep 2022 to Sep 2022. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Berkeley Researchers Replicate DeepSeek R1's Core Tech for Just $30: A Small Mod 1 comment

    • First graph tells the story - below a certain model size (500m params), reinforcement learning is close to useless. Above this (task-dependent) model size threshold, reinforcement learning basically works. I suspect this is what we saw play out with math/coding reasoning models…

      highfrequencyView on HNParent context

  2. OpenAI researchers warned board of AI breakthrough ahead of CEO ouster 1 comment

    • No, that shows only that the dataset is comprised of common problem patterns. The paper explicitly investigates whether memorization has overly impacted performance. From page 10 [1]: > A central question in interpreting Minerva’s solutions is whether performance reflects…

      reasonablekloutView on HNParent context

  3. To build truly intelligent machines, teach them cause and effect 1 comment

    • > There aren't really any current advances outside of sheer scale of input in the models That isn't true. https://ai.googleblog.com/2022/11/characterizing-emergent-ph... While one can correctly argue that this is a consequense of increasing size, no one realized that increasing…

      nlView on HNParent context

  4. Building a Virtual Machine Inside ChatGPT 1 comment

    • So this shows you have a basic understanding of how CNNs represent data for classification tasks. This is excellent! However, you might not be aware of the current research into "Emergent Phenomena in Large Language Models"[1]. Work such as PaLM[2] show that language models…

      nlView on HNParent context

  5. AI language models are struggling to “get” math 1 comment

    • This might make sense as a response solely to the title of the article, but I have to admit I find it puzzling as a reaction to its content. Notwithstanding the title, the article mentions a model called Minerva that scored fully 50% on the MATH dataset of high-school/undergrad…

      edouard-harrisView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.