MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation · HackerPapers discussions

MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation — mentioned by moyix

Thu, 22 Dec 2022 20:46:41 GMT

Based on the reverse engineering done by Parth Thakkar [1], the model used by Copilot is probably about 10x as large (12B parameters), so I would expect Copilot to still win pretty handily (especially since the Codex models are generally a lot better trained than Salesforce CodeGen or InCoder). It's also a little bit hard to compare directly because as Parth documents, there are a lot of extra smarts that go into Copilot on the client side. The SantaCoder paper does have some benchmarks on MultiPL-E though, so you could compare them to the Codex results on that benchmark reported here (but…

Paper and discussion history. A mention is not an endorsement.

MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation — mentioned by emeryberger

Sat, 20 Aug 2022 19:59:34 GMT

Hi all - article author here. A reader of my article sent me this note, pointing me to some relevant hot-off-the-presses work (reproduced with permission): ----- A group recently evaluated the performance of Codex (and another model) on 18 programming languages: https://nuprl.github.io/MultiPL-E/ The high-order bits are: 1. The latest and greatest Codex is now twice as good on its own benchmark suite than the original version published a year ago. 2. It's just as good on Python as it is on JS, Scala, C++, Swift, TypeScript... and other languages are not too far behind. It's not bad at bash of…

Paper and discussion history. A mention is not an endorsement.