HackerPapers

arXiv:2605.03546cs.SE

ProgramBench: Can Language Models Rebuild Programs From Scratch?

John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, Ofir Press

Abstract

Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95\% of tests on only 3\% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.

arXiv comments
Journal reference
DOI
Subjects
cs.SESoftware Engineering cs.AIArtificial Intelligence

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 2 threads, from May 2026 to Oct 2026. Found in 1 HN submission from May 2026 to May 2026. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. ProgramBench: Can Language Models Rebuild Programs from Scratch? 1 comment

    • ScholarlyArticle: "ProgramBench: Can Language Models Rebuild Programs From Scratch?" (2026) https://arxiv.org/abs/2605.03546 Is it that agents don't do the human-in-the-loop engineering management tasks like refactoring and docs and stopping to push for branch coverage before…

      westurnerView on HNParent context

  2. Through the looking glass of benchmark hacking 1 comment

    • I have found it: https://news.ycombinator.com/item?id=48045174 The study paper: https://arxiv.org/abs/2605.03546 Look at Table 3, where the cheating rates of Claude Sonnet, Claude Opus and Gemini were between 20% and 36%, during the coding benchmarks.

      adrian_bView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.