Evaluating Large Language Models Trained on Code · HackerPapers discussions

Evaluating Large Language Models Trained on Code — mentioned by kadushka

Thu, 10 Jul 2025 01:03:17 GMT

Codex paper confirms that GPT-3 could not do any coding tasks. It's right there in the abstract: https://arxiv.org/abs/2107.03374

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by imenani

Wed, 23 Apr 2025 19:24:20 GMT

They fix the temperature at T=0.6 for all k for all models, even though their own Figure 10 shows that RL model benefits from higher temperatures. I would buy the overall claim much more if they swept of temperature parameter for each k and model like they did in the Codex paper [1]. [1] https://arxiv.org/abs/2107.03374

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by moyix

Fri, 14 Feb 2025 14:22:18 GMT

One thing that is interesting is that this was anticipated by the OpenAI Codex paper (which led to GitHub Copilot) all the way back in 2021: > Users might be more inclined to accept the Codex answer under the assumption that the package it suggests is the one with which Codex will be more helpful. As a result, certain players might become more entrenched in the package market and Codex might not be aware of new packages developed after the training data was originally gathered. Further, for already existing packages, the model may make suggestions for deprecated methods. This could increase…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by kristianp

Sat, 27 Apr 2024 03:09:58 GMT

Humaneval was developed for codex I believe: https://arxiv.org/abs/2107.03374

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by godelski

Tue, 06 Feb 2024 18:39:10 GMT

Does anyone know how much spoilage are in these datasets? Common crawl has a lot of websites in it, including Reddit and Stack*. I'm certain there are lots of questions in those datasets and we want to differentiate recall from problem solving (often confused). I have a deep distrust when using large datasets like this given a common one with 60 authors assumed writing leet code style programs by hand would mean they wouldn't appear in the training data (github) and didn't even bother to check. It's really hard to sanitize datasets of this size and deduplication is a much harder task than…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by godelski

Tue, 31 Oct 2023 21:47:25 GMT

What data did you use to train and how do you evaluate your model for overfitting? I ask due to the issues with the HumanEval dataset. ------------- For those that are unfamiliar with the issues, allow me to elaborate. You can find the dataset in the parent's link or here[0] and you can find the paper here[1]. I'll quote from the paper. First is page 2 right above the github link and second is page 4 section 2.2 (note, this paper has 58 authors... 58) > To accurately benchmark our model, we create a dataset of 164 original programming problems with unit tests. These problems assess language…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by swyx

Wed, 06 Sep 2023 18:31:13 GMT

i highlighted this last night which seems to be making the rounds - https://twitter.com/swyx/status/1699369076529971545 Alec's CV (https://www.linkedin.com/in/alecradford/) seems to be: - 2011-2016 BSc Eng from Olin College - 2013+ started a data/AI consultancy as a sophomore that turned into a vague startup/product? - 2015 first paper on GANs coauthored with soumith https://arxiv.org/pdf/1511.06434.pdf%C3 - 2016 first GAN paper under openai email https://proceedings.neurips.cc/paper_files/paper/2016/file/8... - 2017 the generative reviews paper mentioend in the Wired article, with Ilya…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by swyx

Mon, 04 Sep 2023 20:11:16 GMT

> given only one chance to solve it my understanding is that there are 2 usages of the pass@{number} syntax. the HumanEval/Codex paper interprets the {number} as number of attempts[0]. however language modelers seem to use it to denote the number of few shot example demonstrations given in the context. these are starkly different and i wish the syntax wasnt overloaded --- [0] https://arxiv.org/pdf/2107.03374.pdf > Kulal et al. (2019) evaluate functional correctness using the pass@k metric, where k code samples are generated per problem, a problem is considered solved if any sample passes the…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by YeGoblynQueenne

Sun, 02 Jul 2023 20:59:03 GMT

>> - Image descriptions eg. Be My Eyes: https://www.bemyeyes.com/ Not working yet. From the website: Starting today, you can register for the waitlist in the Be My Eyes app. >> - Customer service chat triage. More specifically? Which company is currently using LLMs for this purpose? >> - helping students learn- eg khan academy, tutor Lily Helping them learn, but what? Per wikipedia: Statements made in certain mathematics and physics videos have been questioned for their technical accuracy.[43] Sounds like using LLMs will just generate more garbage teaching material. >> - Helping create and…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by godelski

Wed, 03 May 2023 18:47:19 GMT

My favorite line from the HumanEval paper[0] > It is important for these tasks to be hand-written, since our models are trained on a large fraction of GitHub, which already contains solutions to problems from a variety of sources. So to answer your question, yes, the evaluation dataset is spoiled. You can find such unique and never before seen docstrings like > For a given list of input numbers calculate the Mean Absolute Deviation around the mean of this dataset. Mean Absolute Deviation is the absolute difference between each element and a centerpoint (mean in this case)[1] And here's a repo…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — HN submission

Fri, 27 Jan 2023 21:27:58 GMT

Evaluating Large Language Models Trained on Code

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by BeefWellington

Mon, 17 Oct 2022 03:40:11 GMT

> Well probably no, they didn't pick and choose at all, they just "chose" everyone who put code online with a license. Which is a legal statement of ownership by each of those people, and implies legal liability as well. What you're describing is a choice. They chose which people to believe, with zero vetting. > The point is that with ML training data, such a vast quantity is required that it's unreasonable to expect humans to be able to research and guarantee the legal provenance of it all. I'm not sure what you're presenting here is actually true. A key part of ML training is the training…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by YeGoblynQueenne

Sun, 07 Aug 2022 13:07:26 GMT

Where is this claim from? The Codex paper claims 28.8% on OpenAI's novel dataset: On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. https://arxiv.org/abs/2107.03374 The numbers I quote (5-7%) are on the APPS dataset, reported by SalesForce: https://arxiv.org/abs/2207.01780 (Recently shared on HN). Note that there is no standard dataset like ImageNet or MNIST in program synthesis so it's not correct to speak of a "benchmark". LLM code…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by p1esk

Sun, 24 Jul 2022 17:19:31 GMT

On a dataset of 164 programming problems: https://arxiv.org/pdf/2107.03374.pdf (Section 2.2). The dataset is here: https://github.com/openai/human-eval

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by YeGoblynQueenne

Fri, 15 Jul 2022 10:59:35 GMT

Generating entire functions has been possible for a long time before Copilot. The discipline that studies algorithms that generate code or programs is called Program Synthesis. For a recent review see Gulwani et al: Program Synthesis https://www.semanticscholar.org/paper/Program-Synthesis-Gulw... Copilot and similar systems (like DeepMind's AlphaCode) are a step backwards compared with earlier systems most of which are capable of not only generating code, but generating correct programs, where correctness is measured with respect to a specification, which can be complete or incomplete (as in…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by YeGoblynQueenne

Sat, 08 Jan 2022 22:58:38 GMT

Well, they do say very clearly that they "solve" problems by program synthesis and what they describe is perfectly legit program synthesis. To clarify, program synthesis (or automatic programming) is the task of generating programs from specifications. There are two kinds of program synthesis: deductive program synthesis, from a complete specification of the target program; and inductive program synthesis, or program induction, from an incomplete specification (such as sets of program inputs and outputs, or traces). An example of deductive program synthesis is the generation of low-level code…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by sailingparrot

Thu, 16 Dec 2021 20:26:13 GMT

> I know that at least on most common performance benchmarks these claims are measurably false What "these claims" are you referring to? It seems you are taking issue with only one specific claim of my comment, namely than GPT-3 6B is better quality than GPT-J 6B. Evaluations run by Eulether folks are available here [1] and I have the opposite subjective experience from you. But even assuming I'm wrong, that doesn't change at all the substance of what I am saying: If you need better quality than GPT-J, then GPT-3 (DaVicing, 175B) is your only option. And if you care about latency, last time I…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by YeGoblynQueenne

Thu, 02 Dec 2021 10:53:22 GMT

>> This is also something I'm curious about regarding Copilot. I mentioned that "GitHub Copilot has a list of possible solutions to code completions, so I wonder if it’s just luck that the suggested solution was the correct one". The Codex model on which Copilot is based had about 30% accuracy on the first solution to a coding problem, but 70% when it was allowed to generate 100 solutions and choose the one that passed unit tests. On the other hand, when the best-of-100 solution was chosen according to the probability assigned to it by the model it scored 45% [1]. So it's kiind of luck.…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by moyix

Sat, 04 Sep 2021 15:43:39 GMT

Yes, they mention this in the Codex paper [1]: > Since Codex is evaluated on natural language prompts, we hypothesized that it would be beneficial to fine-tune from the GPT-3 (Brown et al., 2020) model family, which already contains strong natural language representations. Surprisingly, we did not observe improvements when starting from a pre-trained language model, possibly because the finetuning dataset is so large. Nevertheless, models fine-tuned from GPT converge more quickly, so we apply this strategy for all subsequent experiments. It's not completely clear exactly what relationship the…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by arthur2e5

Fri, 03 Sep 2021 17:37:08 GMT

GitHub Copilot is literally GPT adapted for code. The paper on OpenAI Codex, the stuff powering Copilot, makes it very clear in the abstract. https://arxiv.org/abs/2107.03374 Either you call both AI or you call neither AI. (A previous version of the comment stated that it was tuned from GPT-3. This is incorrect; the simpler GPT was used for faster convergence.)

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by lstmemery

Tue, 10 Aug 2021 20:52:58 GMT

I have to disagree with you here. In the Codex paper[1], they have two datasets that Codex got correct about 3% of the time. These are interview and code competition questions. From the paper: "Indeed, a strong student who completes an introductory computer science course is expected to be able to solve a larger fraction of problems than Codex-12B." This suggests to me that Codex really doesn't understand anything about the language beyond syntax. I have no doubt that future systems will improve on this benchmark, but they will likely take advantage of the AST and could use unit tests in a…

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — HN submission

Fri, 09 Jul 2021 17:39:30 GMT

Evaluating Large Language Models Trained on Code

12 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Evaluating Large Language Models Trained on Code — mentioned by devetec

Fri, 09 Jul 2021 13:15:54 GMT

There's an entire academic paper detailing exactly how it works. https://arxiv.org/abs/2107.03374

Paper and discussion history. A mention is not an endorsement.

Evaluating Large Language Models Trained on Code — HN submission

Thu, 08 Jul 2021 21:10:26 GMT

Evaluating Large Language Models Trained on Code (paper about GH copilot model)

4 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Evaluating Large Language Models Trained on Code — HN submission

Thu, 08 Jul 2021 12:20:59 GMT

Evaluating Large Language Models Trained on Code(GitHub Copilot)

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Evaluating Large Language Models Trained on Code — HN submission

Thu, 08 Jul 2021 00:36:26 GMT

Evaluating Large Language Models Trained on Code

11 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.