HackerPapers

arXiv:2310.17680cs.SE

CodeFusion: A Pre-trained Diffusion Model for Code Generation

Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, Gust Verbruggen

Abstract

Imagine a developer who can only change their last line of code, how often would they have to start writing a function from scratch before it is correct? Auto-regressive models for code generation from natural language have a similar limitation: they do not easily allow reconsidering earlier tokens generated. We introduce CodeFusion, a pre-trained diffusion code generation model that addresses this limitation by iteratively denoising a complete program conditioned on the encoded natural language. We evaluate CodeFusion on the task of natural language to code generation for Bash, Python, and Microsoft Excel conditional formatting (CF) rules. Experiments show that CodeFusion (75M parameters) performs on par with state-of-the-art auto-regressive systems (350M-175B parameters) in top-1 accuracy and outperforms them in top-3 and top-5 accuracy due to its better balance in diversity versus quality.

arXiv comments
Contains inappropriately sourced conjecture of OpenAI's ChatGPT parameter count from www.forbes.com/sites/forbestechcouncil/2023/02/17/is-bigger-better-why-the-chatgpt-vs-gpt-3-vs-gpt-4-battle-is-just-a-family-chat, a citation which was omitted. The authors do not have direct knowledge or verification of this information, and relied solely on this article, which may lead to public confusion
Journal reference
DOI
Subjects
cs.SESoftware Engineering cs.AIArtificial Intelligence cs.CLComputation and Language cs.PLProgramming Languages

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 3 threads, from Oct 2023 to Mar 2024. Found in 2 HN submissions from Oct 2023 to Oct 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. Emad Mostaque resigned as CEO of Stability AI 1 comment

    • To be precise, ChatGPT 3.5 turbo being 20B is officially a mistake from a Microsoft Researcher, quoting a wrong source published before the release of chatgpt3.5 turbo. Up to you to believe it or not. But I wouldn’t claim it’s a 20B according to Microsoft Researchers. The…

      speedgooseView on HNParent context

  2. Orca 2: Teaching Small Language Models How to Reason 1 comment

    • The paper was https://arxiv.org/abs/2310.17680 It has been withdrawn with this note: > Contains inappropriately sourced conjecture of OpenAI's ChatGPT parameter count from this http URL, a citation which was omitted. The authors do not have direct knowledge or verification of…

      nlView on HNParent context

  3. Microsoft says GPT 3.5 has 20B parameters? 2 comments

    • Link to the paper directly https://arxiv.org/pdf/2310.17680.pdf The system that they describe, Codefusion is interesting because it's a diffusion model for generating code rather than an autoregressive model like most LLM code generators.

      Flux159View on HNParent context

    • Also on HuggingFace: https://huggingface.co/papers/2310.17680 > We evaluate CodeFusion on the task of natural language to code generation for Bash, Python, and Microsoft Excel conditional formatting (CF) rules. Microsoft Excel conditional formatting, really? The future we want…

      airstrikeView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.