Measuring Mathematical Problem Solving With the MATH Dataset · HackerPapers discussions

Measuring Mathematical Problem Solving With the MATH Dataset — mentioned by tadamcz

Thu, 30 Jan 2025 20:59:59 GMT

Hi! I'm Tom, a machine learning engineer at the nonprofit research institute Epoch AI [0]. I've been working on building infrastructure to: * run LLM evaluations systematically and at scale * share the data with the public in a rigorous and transparent way We use the UK government's Inspect [1] library to run the evaluations. As soon as I saw this news on HN, I evaluated Mistral Small 3 on MATH [2] level 5 (hardest subset, 1,324 questions). I get an accuracy of 0.45 (± 0.011). We sample the LLM 8 times for each question, which lets us obtain less noisy estimates of mean accuracy, and measure…

Paper and discussion history. A mention is not an endorsement.

Measuring Mathematical Problem Solving With the MATH Dataset — mentioned by haltist

Tue, 17 Oct 2023 00:50:32 GMT

Part of an abstract from referenced articles: > Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the…

Paper and discussion history. A mention is not an endorsement.

Measuring Mathematical Problem Solving With the MATH Dataset — mentioned by nl

Thu, 30 Mar 2023 06:02:03 GMT

The most relevant reported benchmark results are on https://arxiv.org/abs/2103.03874 They compare it against PaLM and Minerva, both of which are fine tuned on math datasets. It outperforms both at roughly the same parameter count (eg Llama 65B vs Palm and Minerva 62B) but unclear how much of this is due to encoding vs the many other differences. It is useful to see that the performance increase is clearly not due to fine turning though.

Paper and discussion history. A mention is not an endorsement.

Measuring Mathematical Problem Solving With the MATH Dataset — HN submission

Wed, 10 Nov 2021 09:00:47 GMT

Measuring mathematical problem solving with the MATH dataset

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.