CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning · HackerPapers discussions

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning — mentioned by YeGoblynQueenne

Sun, 07 Aug 2022 13:07:26 GMT

Where is this claim from? The Codex paper claims 28.8% on OpenAI's novel dataset: On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. https://arxiv.org/abs/2107.03374 The numbers I quote (5-7%) are on the APPS dataset, reported by SalesForce: https://arxiv.org/abs/2207.01780 (Recently shared on HN). Note that there is no standard dataset like ImageNet or MNIST in program synthesis so it's not correct to speak of a "benchmark". LLM code…

Paper and discussion history. A mention is not an endorsement.

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning — mentioned by YeGoblynQueenne

Sun, 07 Aug 2022 12:47:30 GMT

>> Do you remember how people on HN used to take comfort that AI couldn't write code? Now they're taking comfort in the fact that AI can't write code well. If you're referring to neural program synthesis with large language models (LLMs), the performance of current systems is pitifully low, with best results in the range of 5-7% correct programs when evaluating on a test set similar to the one used for training (and allowing a single "guess"; it goes up to 30% with 100 guesses; happy to point to references if required). So no, "AI" (meaning deep learning I presume) cannot write code very…

Paper and discussion history. A mention is not an endorsement.

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning — mentioned by moyix

Thu, 28 Jul 2022 19:10:07 GMT

Reposting my commentary: The TL;DR is that this appears to beat both OpenAI Codex (12 billion params) and DeepMind's AlphaCode (1 billion params), despite having only 770M parameters. There is also a paper: https://arxiv.org/abs/2207.01780 And the code is available: https://github.com/salesforce/CodeRL So this seems to buck the trend of needing enormous models, by being smarter about taking into account the output from running the AI-generated program on test cases.

Paper and discussion history. A mention is not an endorsement.

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning — mentioned by moyix

Thu, 21 Jul 2022 23:44:38 GMT

The TL;DR is that this appears to beat both OpenAI Codex (12 billion params) and DeepMind's AlphaCode (1 billion params), despite having only 770M parameters. There is also a paper: https://arxiv.org/abs/2207.01780 And the code is available: https://github.com/salesforce/CodeRL

Paper and discussion history. A mention is not an endorsement.