HackerPapers

arXiv:2306.08997cs.CL

Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models

Sarah J. Zhang, Samuel Florin, Ariel N. Lee, Eamon Niknafs, Andrei Marginean, Annie Wang, Keith Tyser, Zad Chin, Yann Hicke, Nikhil Singh, Madeleine Udell, Yoon Kim, Tonio Buonassisi, Armando Solar-Lezama, Iddo Drori

Abstract

We curate a comprehensive dataset of 4,550 questions and solutions from problem sets, midterm exams, and final exams across all MIT Mathematics and Electrical Engineering and Computer Science (EECS) courses required for obtaining a degree. We evaluate the ability of large language models to fulfill the graduation requirements for any MIT major in Mathematics and EECS. Our results demonstrate that GPT-3.5 successfully solves a third of the entire MIT curriculum, while GPT-4, with prompt engineering, achieves a perfect solve rate on a test set excluding questions based on images. We fine-tune an open-source large language model on this dataset. We employ GPT-4 to automatically grade model responses, providing a detailed performance breakdown by course, question, and answer type. By embedding questions in a low-dimensional space, we explore the relationships between questions, topics, and classes and discover which questions and classes are required for solving other questions and classes through few-shot learning. Our analysis offers valuable insights into course prerequisites and curriculum design, highlighting language models' potential for learning and improving Mathematics and EECS education.

arXiv comments
Did not receive permission to release the data or model fine-tuned on the data
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 3 identifiable commenters in 3 comments across 2 threads, from Jun 2023 to Jun 2023. Found in 2 HN submissions from Jun 2023 to Jun 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. On the paper “Exploring the MIT Mathematics and EECS Curriculum Using LLMs” [pdf] 2 comments

    • https://arxiv.org/abs/2306.08997 “arXiv will not consider removal for reasons such as journal similarity detection, nor failure to obtain consent from co-authors, as these do not invalidate the license applied by the submitter” The submitter can mark the paper as “withdrawn” but…

      dfdzView on HNParent context

    • Original paper is on arxiv: https://arxiv.org/abs/2306.08997

      jlaneveView on HNParent context

  2. No, GPT4 Can’t Ace MIT 1 comment

    • I just looked again and I didn't see that claim, can you verify? https://arxiv.org/pdf/2306.08997.pdf If as per the linked critique, some of the questions in the test set were basically nonsense, then clearly they couldn't have manually verified all the answers or they would…

      mquanderView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.