Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation · HackerPapers discussions

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation — mentioned by EvgeniyZh

Sat, 26 Aug 2023 06:29:24 GMT

I have a several arguments why contamination is probably not the main reason of performance difference. When we worked on StarCoder, people ran gpt-4 on MultiPL-E, which doesn't have canonical solutions in the internet, and the performance was higher that what you would expect from official numbers Official contamination analysis shows only minor drop in performance even though contamination is fairly high (you may argue that contamination is higher now or that rlhf has stronger effect) There is significant drop in performance when testing on HumanEval+ [1], which shouldn't happen if model…

Paper and discussion history. A mention is not an endorsement.

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation — mentioned by EvgeniyZh

Thu, 24 Aug 2023 16:28:57 GMT

Note that current GPT-4 pass@1 for HumanEval is closer to 90% than to 67% reported in GPT-4 technical report, as reported, e.g., in [1] [1] https://arxiv.org/abs/2305.01210

Paper and discussion history. A mention is not an endorsement.

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation — mentioned by lhl

Wed, 19 Jul 2023 14:21:28 GMT

I have not read the paper yet (in my backlog, here's the paper: https://arxiv.org/pdf/2307.09009.pdf), but it's important note that the paper is entitled "How Is ChatGPT’s Behavior Changing over Time?" not that it's necessarily "getting worse." Here's a more nuanced (not an AI clout chasing account) discussion by Arvind Narayanan (Princeton CS prof) about the results: https://twitter.com/random_walker/status/1681489529494970368 One thing that I have confirmed is while the abstract and intro talk about evaluating "code generation" as if GPT-4 code generation is getting worse, In is 3.3/Figure…

Paper and discussion history. A mention is not an endorsement.

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation — mentioned by famouswaffles

Fri, 14 Jul 2023 22:14:06 GMT

GPT-4 out in the wild's (reproducible) performance appears to be much higher than 67. Testing from 3/15 (presumably on the 0314 model) seems to be at 85.36% (https://twitter.com/amanrsanger/status/1635751764577361921). And the linked paper from my post(https://doi.org/10.48550/arXiv.2305.01210) got a pass@1 of 88.4 from GPT-4 recently (May? June?).

Paper and discussion history. A mention is not an endorsement.

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation — mentioned by lhl

Tue, 11 Jul 2023 14:43:46 GMT

While that's what the Technical Report (https://arxiv.org/pdf/2303.08774v3.pdf) says, but GPT-4 out in the wild's (reproducible) performance appears to be much higher now. Testing from 3/15 (presumably on the 0314 model) seems to be at 85.36% (https://twitter.com/amanrsanger/status/1635751764577361921). And the linked paper from my post(https://doi.org/10.48550/arXiv.2305.01210) got a pass@1 of 88.4 from GPT-4 recently (May? June?). Out of curiousity, I was trying out gpt-4-0613 and claude-v2 with https://github.com/getcursor/eval, but sadly I'm getting hangs at 3% with both of them (maybe…

Paper and discussion history. A mention is not an endorsement.

Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation — mentioned by lhl

Tue, 11 Jul 2023 14:12:14 GMT

Since I've been on a AI code-helper kick recently. According to the post, Claude 2 now 71.2%, a significant upgrade from 1.3 (56.0%). (Found in model card: pass@1) For comparison: * GPT-4 claims 85.4 on HumanEval, in a recent paper https://arxiv.org/pdf/2303.11366.pdf GPT-4 was tested at 80.1 pass@1 and 91 pass@1 using their Reflexion technique. They also include MBPP and Leetcode Hard benchmark comparisons * WizardCoder, a StarCoder fine-tune is one of the top open models, scoring a 57.3 pass@1, model card here: https://huggingface.co/WizardLM/WizardCoder-15B-V1.0 * The best open model I…

Paper and discussion history. A mention is not an endorsement.