AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators · HackerPapers discussions

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators — mentioned by moritz

Fri, 25 Sep 2026 15:33:15 GMT

You‘ll notice a lot of "I found"s and "my hunch"s and similar in the comments. There aren’t many comparative benchmarks, and obviously the design of such a benchmark is difficult, but, e.g. https://arxiv.org/abs/2508.09101 In this benchmark, models can correctly solve Rust problems 61% on first pass — A far cry from other languages such as C# (88%) or Elixir (97%, no static typing whatsoever).

Paper and discussion history. A mention is not an endorsement.

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators — mentioned by Stromgren

Sat, 18 Jul 2026 17:25:56 GMT

In fact it came out as the absolute best in a comparison by Tencent (1). See table 4. It’s more than a year old though. 1. https://arxiv.org/pdf/2508.09101

Paper and discussion history. A mention is not an endorsement.

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators — mentioned by skhameneh

Thu, 16 Jul 2026 21:37:49 GMT

I like Rust and I'm an advocate of Rust, but this really isn't true (at least it hasn't been and I doubt much has significantly changed). The syntax complexity and the ecosystem haven't been ideal for LLM development. And there have been publications on findings of LLM efficacy with different languages. Rust is most often towards the lower end of efficiency/correctness when benchmarked. https://arxiv.org/html/2508.09101v1

Paper and discussion history. A mention is not an endorsement.

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators — mentioned by briandw

Tue, 12 May 2026 19:18:51 GMT

In this paper AutoCodeBench https://arxiv.org/pdf/2508.09101 LLMs seemed to do best on the strongly typed and functional language Elixir. This is surprising since there isn't very much training data. However the examples that it's seen are usually high quality and there aren't a large number of different way of doing the same thing in Elixir.

Paper and discussion history. A mention is not an endorsement.

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators — mentioned by avereveard

Mon, 11 May 2026 21:16:33 GMT

https://arxiv.org/pdf/2508.09101 tldr 2% average point lost on Rust compared to python, gap vary by model, go has a better upper bound but opus had it 3% below python. benchmark is a bit old but research on why is there, article is just vibes

Paper and discussion history. A mention is not an endorsement.

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators — mentioned by hashmap

Fri, 24 Apr 2026 15:36:54 GMT

> The language they do best with is the one with the largest corpus in the training set. Not the case, strangely. They do best with Elixir. https://arxiv.org/pdf/2508.09101

Paper and discussion history. A mention is not an endorsement.

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators — mentioned by moritz

Mon, 02 Mar 2026 19:36:30 GMT

https://arxiv.org/abs/2508.09101 In this benchmark, models can correctly solve Rust problems 61% on first pass — A far cry from other languages such as C# (88%) or Elixir (a “buggy dynamic language”) where they perform best (97%). I wonder why that is, it’s quite surprising. Obviously details of their benchmark design matter, but this study doesn’t support your claims.

Paper and discussion history. A mention is not an endorsement.