HackerPapers

arXiv:2107.03374cs.LG

Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba

Abstract

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

arXiv comments
corrected typos, added references, added authors, added acknowledgements
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 13 identifiable commenters in 21 comments across 21 threads, from Jul 2021 to Jul 2025. Found in 5 HN submissions from Jul 2021 to Jan 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments21 threads

  1. Mercury: Ultra-fast language models based on diffusion 1 comment

    • Codex paper confirms that GPT-3 could not do any coding tasks. It's right there in the abstract: https://arxiv.org/abs/2107.03374

      kadushkaView on HNParent context

  2. Does RL Incentivize Reasoning in LLMs Beyond the Base Model? 1 comment

    • They fix the temperature at T=0.6 for all k for all models, even though their own Figure 10 shows that RL model benefits from higher temperatures. I would buy the overall claim much more if they swept of temperature parameter for each k and model like they did in the Codex paper…

      imenaniView on HNParent context

  3. AI is stifling new tech adoption? 1 comment

    • One thing that is interesting is that this was anticipated by the OpenAI Codex paper (which led to GitHub Copilot) all the way back in 2021: > Users might be more inclined to accept the Codex answer under the assumption that the package it suggests is the one with which Codex…

      moyixView on HNParent context

  4. Qwen1.5-110B 1 comment
  5. DeepSeekMath 7B achieved 51.7% on MATH benchmark 1 comment

    • Does anyone know how much spoilage are in these datasets? Common crawl has a lot of websites in it, including Reddit and Stack*. I'm certain there are lots of questions in those datasets and we want to differentiate recall from problem solving (often confused). I have a deep…

      godelskiView on HNParent context

  6. Phind Model beats GPT-4 at coding, with GPT-3.5 speed and 16k context 1 comment

    • What data did you use to train and how do you evaluate your model for overfitting? I ask due to the issues with the HumanEval dataset. ------------- For those that are unfamiliar with the issues, allow me to elaborate. You can find the dataset in the parent's link or here[0] and…

      godelskiView on HNParent context

  7. What OpenAI really wants 1 comment

    • i highlighted this last night which seems to be making the rounds - https://twitter.com/swyx/status/1699369076529971545 Alec's CV (https://www.linkedin.com/in/alecradford/) seems to be: - 2011-2016 BSc Eng from Olin College - 2013+ started a data/AI consultancy as a sophomore…

      swyxView on HNParent context

  8. Refact Code LLM: 1.6B LLM for code that reaches 32% HumanEval 1 comment

    • > given only one chance to solve it my understanding is that there are 2 usages of the pass@{number} syntax. the HumanEval/Codex paper interprets the {number} as number of attempts[0]. however language modelers seem to use it to denote the number of few shot example…

      swyxView on HNParent context

  9. GPT-Migrate converts repos from one lang/framework to another 1 comment

    • >> - Image descriptions eg. Be My Eyes: https://www.bemyeyes.com/ Not working yet. From the website: Starting today, you can register for the waitlist in the Be My Eyes app. >> - Customer service chat triage. More specifically? Which company is currently using LLMs for this…

      YeGoblynQueenneView on HNParent context

  10. Replit's new Code LLM: Open Source, 77% smaller than Codex, trained in 1 week 1 comment

    • My favorite line from the HumanEval paper[0] > It is important for these tasks to be hand-written, since our models are trained on a large fraction of GitHub, which already contains solutions to problems from a variety of sources. So to answer your question, yes, the evaluation…

      godelskiView on HNParent context

  11. GitHub Copilot, with “public code” blocked, emits my copyrighted code 1 comment

    • > Well probably no, they didn't pick and choose at all, they just "chose" everyone who put code online with a license. Which is a legal statement of ownership by each of those people, and implies legal liability as well. What you're describing is a choice. They chose which…

      BeefWellingtonView on HNParent context

  12. AlphaFold won’t revolutionise drug discovery 1 comment

    • Where is this claim from? The Codex paper claims 28.8% on OpenAI's novel dataset: On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J…

      YeGoblynQueenneView on HNParent context

  13. A new way to predict when software jobs will become automated 1 comment

    • On a dataset of 164 programming problems: https://arxiv.org/pdf/2107.03374.pdf (Section 2.2). The dataset is here: https://github.com/openai/human-eval

      p1eskView on HNParent context

  14. ML code generation vs. coding by hand: what we think programming will look like 1 comment

    • Generating entire functions has been possible for a long time before Copilot. The discipline that studies algorithms that generate code or programs is called Program Synthesis. For a recent review see Gulwani et al: Program Synthesis…

      YeGoblynQueenneView on HNParent context

  15. A neural network solves and generates mathematics problems by program synthesis 1 comment

    • Well, they do say very clearly that they "solve" problems by program synthesis and what they describe is perfectly legit program synthesis. To clarify, program synthesis (or automatic programming) is the task of generating programs from specifications. There are two kinds of…

      YeGoblynQueenneView on HNParent context

  16. GPT-3 APIs are now in public beta 1 comment

    • > I know that at least on most common performance benchmarks these claims are measurably false What "these claims" are you referring to? It seems you are taking issue with only one specific claim of my comment, namely than GPT-3 6B is better quality than GPT-J 6B. Evaluations…

      sailingparrotView on HNParent context

  17. Solving Advent of Code Puzzles with GitHub Copilot: Day 1 1 comment

    • >> This is also something I'm curious about regarding Copilot. I mentioned that "GitHub Copilot has a list of possible solutions to code completions, so I wonder if it’s just luck that the suggested solution was the correct one". The Codex model on which Copilot is based had…

      YeGoblynQueenneView on HNParent context

  18. Z3 approach to discover that “q_rsqrt” is in Copilot's slur list 1 comment

    • Yes, they mention this in the Codex paper [1]: > Since Codex is evaluated on natural language prompts, we hypothesized that it would be beneficial to fine-tune from the GPT-3 (Brown et al., 2020) model family, which already contains strong natural language representations.…

      moyixView on HNParent context

  19. GitHub Copilot Generated Insecure Code in 40% of Circumstances During Experiment 1 comment

    • GitHub Copilot is literally GPT adapted for code. The paper on OpenAI Codex, the stuff powering Copilot, makes it very clear in the abstract. https://arxiv.org/abs/2107.03374 Either you call both AI or you call neither AI. (A previous version of the comment stated that it was…

      arthur2e5View on HNParent context

  20. OpenAI Codex 1 comment

    • I have to disagree with you here. In the Codex paper[1], they have two datasets that Codex got correct about 3% of the time. These are interview and code competition questions. From the paper: "Indeed, a strong student who completes an introductory computer science course is…

      lstmemeryView on HNParent context

  21. All public GitHub code was used in training Copilot 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.