arXiv:2301.13867cs.LG2023-01-31

Mathematical Capabilities of ChatGPT

Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Julius Berner

Abstract

We investigate the mathematical capabilities of two iterations of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel methodology. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., the Lean Mathematical Library), current datasets of natural-language mathematics, used to benchmark language models, either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets also test whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a mathematical search engine and knowledge base interface. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if your goal is to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!

arXiv comments: Added further evaluations on another ChatGPT version and on GPT-4. The GHOSTS and miniGHOSTS datasets are available at https://github.com/xyfrieder/science-GHOSTS
Journal reference: NeurIPS 2023 Datasets and Benchmarks
DOI
Subjects: cs.LGMachine Learning cs.AIArtificial Intelligence cs.CLComputation and Language

On Hacker News

Linked by 5 identifiable commenters in 10 comments across 10 threads, from Mar 2023 to Jun 2024. Found in 1 HN submission from Feb 2023 to Feb 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments10 threads

Terence Tao on proof checkers and AI programs Jun 20241 comment
- LLMs can do math. AFAIK I know this is the SOTA if you don't formalize it: https://arxiv.org/abs/2301.13867
  
  data_maanJun 13, 2024View on HN Parent context
Elon Musk sues Sam Altman, Greg Brockman, and OpenAI [pdf] Mar 20241 comment
- https://arxiv.org/abs/2301.13867
  
  data_maanMar 2, 2024View on HN Parent context
GPT-4 Understands Oct 20231 comment
- > The implications are vast. We may be able to translate between languages that have never had a “Rosetta Stone”. Any animals that have a true language could have it decoded. And while an LLM that’s gotten an 8 year old’s understanding of balancing assorted items isn’t that…
  
  rvzOct 16, 2023View on HN Parent context
AI is replacing customer service jobs across the globe Oct 20231 comment
- Might be a good idea to keep at least one human in the loop. Otherwise, just use grad-level maths to confuse it into giving refunds I suppose [1]. [1] https://arxiv.org/abs/2301.13867
  
  nprateemOct 4, 2023View on HN Parent context
The Pile: An 800GB dataset of diverse text for language modeling (2020) Jul 20231 comment
- Could you share more about copyright? For example, aren't you worried that now, with all kinds of lawsuits happening [1] and copyright issues that were found in existing datasets [2], that you might get threatening letters from a lawyer some day? I'm the author of [3] where we…
  
  sfriedrJul 11, 2023View on HN Parent context
On the paper “Exploring the MIT Mathematics and EECS Curriculum Using LLMs” [pdf] Jun 20231 comment
- Here's a more rigorous take on evaluating the output of GPT models on math: https://arxiv.org/abs/2301.13867
  
  data_maanJun 25, 2023View on HN Parent context
When it comes to advanced math, ChatGPT is no star student May 20231 comment
- specifically https://arxiv.org/pdf/2301.13867.pdf
  
  082349872349872May 22, 2023View on HN Parent context
[dead] Apr 20231 comment
- In February I published a paper on mathematics + ChatGPT (https://arxiv.org/abs/2301.13867) and my colleagues, who were also on LinkedIn, told me it garnered quite a bit of attention. So I signed up to LinkedIn on 3rd Feb. 2023, thinking that now would be a good time to step out…
  
  sfriedrApr 15, 2023View on HN Parent context
Ask HN: Who wants to be hired? (April 2023) Apr 20231 comment
- Location: Oxford, UK Remote: Preferred Willing to relocate: Depending on location (inside EU preferred, but willing to consider other options depending on the work) Technologies: LLMs, other Deep Learning models (e.g. Vision Transformer), prompt engineering, PyTorch, LLMs…
  
  sfriedrApr 15, 2023View on HN Parent context
Ask HN: Who wants to be hired? (March 2023) Mar 20231 comment
- Location: Oxford, UK Remote: Preferred Willing to relocate: Depending on location (inside EU preferred, but willing to consider other options depending on the work) Technologies: LLMs, other Deep Learning models (e.g. Vision Transformer), prompt engineering, PyTorch, LLMs…
  
  sfriedrMar 4, 2023View on HN Parent context

Submitted to Hacker News

Mathematical Capabilities of ChatGPT2 points0 commentsFeb 2023

Connected through shared HN threads, not inferred similarity or a recommendation.

Abstract

On Hacker News

Discussed in comments10 threads

Submitted to Hacker News

Also discussed alongside