HackerPapers

arXiv:2302.06590cs.SE

The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

Sida Peng, Eirini Kalliamvakou, Peter Cihon, Mert Demirer

Abstract

Generative AI tools hold promise to increase human productivity. This paper presents results from a controlled experiment with GitHub Copilot, an AI pair programmer. Recruited software developers were asked to implement an HTTP server in JavaScript as quickly as possible. The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group. Observed heterogenous effects show promise for AI pair programmers to help people transition into software development careers.

arXiv comments
Journal reference
DOI
Subjects
cs.SESoftware Engineering

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 4 threads, from Jun 2023 to Jan 2026. Found in 3 HN submissions from Mar 2023 to Jul 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. 2025: The Year in LLMs 1 comment

    • That's why I gave you data! METR study was 16 people using Sonnet 3.5/3.7. Data I'm talking about is 10s of thousands of people and is much more up to date. Some counter examples to METR that are in the literature but I'll just say: "rigor" here is very difficult (including…

      aspenmartinView on HNParent context

  2. I tried coding with AI, I became lazy and stupid 1 comment

    • There are more studies out there, but here are a couple I know of offhand, showing a 25% to 55% boost. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4945566 https://arxiv.org/abs/2302.06590 The METR study that you're likely talking about had a lot of nuances that don't get…

      keedaView on HNParent context

  3. AI promised efficiency. Instead, it's making us work harder 1 comment

    • The study that said devs were 19% less productive was flawed in a number of ways: small sample size, poorly chosen tasks, no prior training on AI tools. The science is by no means conclusive. A joint Harvard / MIT study found developers completed tasks up to 50% faster [0]! [0]…

      davelyView on HNParent context

  4. Measuring the productivity impact of generative AI 1 comment

    • > Have there been any good studies on programmer productivity? Yes, but mostly from the companies developing these products: * The CoPilot productivity study by Peng et al. that is summarized in a Blog post [1] with some additional details in the Arxiv preprint [2]. They find…

      NathanFultonView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.