Solving Quantitative Reasoning Problems with Language Models · HackerPapers discussions

Solving Quantitative Reasoning Problems with Language Models — mentioned by highfrequency

Tue, 28 Jan 2025 17:53:40 GMT

First graph tells the story - below a certain model size (500m params), reinforcement learning is close to useless. Above this (task-dependent) model size threshold, reinforcement learning basically works. I suspect this is what we saw play out with math/coding reasoning models - until recently, the base models were not good enough for ~random output search to hit on a correct path with any reasonable frequency. Below this threshold of base model intelligence, the only efficient way forward was to collect plain supervised data (either through human labeled math problem solutions [1] or…

Paper and discussion history. A mention is not an endorsement.

Solving Quantitative Reasoning Problems with Language Models — mentioned by reasonableklout

Thu, 23 Nov 2023 02:10:25 GMT

No, that shows only that the dataset is comprised of common problem patterns. The paper explicitly investigates whether memorization has overly impacted performance. From page 10 [1]: > A central question in interpreting Minerva’s solutions is whether performance reflects genuine analytic capability or instead rote memorization. This is especially relevant as there has been much prior work indicating that language models often memorize some fraction of their training data ... In order to evaluate the degree to which our models solve problems by recalling information memorized from training…

Paper and discussion history. A mention is not an endorsement.

Solving Quantitative Reasoning Problems with Language Models — mentioned by nl

Sun, 26 Feb 2023 23:38:08 GMT

> There aren't really any current advances outside of sheer scale of input in the models That isn't true. https://ai.googleblog.com/2022/11/characterizing-emergent-ph... While one can correctly argue that this is a consequense of increasing size, no one realized that increasing the size of LLMs would give them reasoning ability in 2018. Indeed, before Minerva[1] came out expert forecasters predicted[2] a 12% improvement in the ability on the MATH benchmark[3]. Minerva improved it by 50% and PaLM improved it even more. [1] https://arxiv.org/abs/2206.14858 [2]…

Paper and discussion history. A mention is not an endorsement.

Solving Quantitative Reasoning Problems with Language Models — mentioned by nl

Sun, 04 Dec 2022 01:58:19 GMT

So this shows you have a basic understanding of how CNNs represent data for classification tasks. This is excellent! However, you might not be aware of the current research into "Emergent Phenomena in Large Language Models"[1]. Work such as PaLM[2] show that language models evolve the ability to "reason" over the internal data representation. Minerva explicitly breaks down that ability by "comb[ining] several techniques, including few-shot prompting, chain of thought or scratchpad prompting, and majority voting, to achieve state-of-the-art performance on STEM reasoning tasks." This lets it…

Paper and discussion history. A mention is not an endorsement.

Solving Quantitative Reasoning Problems with Language Models — mentioned by edouard-harris

Thu, 13 Oct 2022 00:25:27 GMT

This might make sense as a response solely to the title of the article, but I have to admit I find it puzzling as a reaction to its content. Notwithstanding the title, the article mentions a model called Minerva that scored fully 50% on the MATH dataset of high-school/undergrad mathematical problems. For comparison, a human computer science PhD student scored 40%. [1] For context, Minerva came out this July. When it was tested on a national math exam, it scored higher than that year's class of graduating high school seniors. [2] A mere eight months (!) earlier, OpenAI had announced [3] they'd…

Paper and discussion history. A mention is not an endorsement.

Solving Quantitative Reasoning Problems with Language Models — HN submission

Tue, 20 Sep 2022 17:59:04 GMT

Solving Quantitative Reasoning Problems with Language Models

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.