HackerPapers

arXiv:1901.10220cs.SE

On the Impact of Programming Languages on Code Quality

Emery D. Berger, Celeste Hollenbeck, Petr Maj, Olga Vitek, Jan Vitek

Abstract

This paper is a reproduction of work by Ray et al. which claimed to have uncovered a statistically significant association between eleven programming languages and software defects in projects hosted on GitHub. First we conduct an experimental repetition, repetition is only partially successful, but it does validate one of the key claims of the original work about the association of ten programming languages with defects. Next, we conduct a complete, independent reanalysis of the data and statistical modeling steps of the original study. We uncover a number of flaws that undermine the conclusions of the original study as only four languages are found to have a statistically significant association with defects, and even for those the effect size is exceedingly small. We conclude with some additional sources of bias that should be investigated in follow up work and a few best practice recommendations for similar efforts.

arXiv comments
Accepted, to appear in TOPLAS (ACM Transactions of Programming Languages and Systems)
Journal reference
DOI
doi:10.1145/3340571
Subjects
cs.SESoftware Engineering

On Hacker News

Linked by 9 identifiable commenters in 10 comments across 9 threads, from Apr 2019 to Dec 2025. Found in 4 HN submissions from Jan 2019 to Feb 2026. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. The programmers who live in Flatland 1 comment

    • If we approach the question as engineers, scientifically, with numbers and studies, not anecdotes and hand-waving, then Clojure is hands down the best language in terms of productivity and bug reduction. To this day, I know of no study that was able to demonstrate superiority of…

      DeadEarnestView on HNParent context

  2. Proposal: Less error-prone loop variable scoping 2 comments

    • > Do you have any info on this? There have been some studies, and while all methodologies have tradeoff's go and clojure sit near the bottom of this list in bugs/committ: https://arxiv.org/pdf/1901.10220.pdf Which is fascinating because they are very different languages whose…

      zbobet2012View on HNParent context

    • > all methodologies have tradeoff's go and clojure sit near the bottom of this list in bugs/committ: https://arxiv.org/pdf/1901.10220.pdf You have linked a study which, if you read the abstract, effectively claims "we tried to reproduce a study about…

      TheDongView on HNParent context

  3. Why We Switched from Python to Go (2021) 1 comment

    • There's a speculation in the first citation: > Python is not maintanable above 50k to 100k lines of code and because of that people consider this code bases very large Note that this is a speculation, not a conclusion, the article is not very thorough, and LOC is mostly used…

      klodolphView on HNParent context

  4. Immutability Changes Everything (2016) 1 comment

    • First of all, I am arguing for the base case: absent any studies, the null hypothesis is naturally that there are no major differences between programming paradigms. On the issue of bug counts, there was one huge study, that was successfully reproduced [0], which finds that…

      tsimionescuView on HNParent context

  5. Functional Programming in Go with Generics 1 comment

    • > > it enhances the ability to reason about code > In your opinion. Correct. But my opinion is informed by years of experience with functional programming, procedural programming, and OOP (each). > Please provide any evidence that this is true, especially the ‘safe’ part. Any…

      ebingdomView on HNParent context

  6. Haskell is our first choice for building production software systems 1 comment

    • >Is there really a formal proof or Software Engineering paper that proves this? Yes (page 7): https://arxiv.org/pdf/1901.10220.pdf

      nix23View on HNParent context

  7. What I Want from a Type System (2016) 1 comment

    • Sure, however nobody has been shown that one approach is more costly than the other in practice. The most recent research available is the replication of the large scale GitHub study [1]. The relative effects of language choice on code quality is less than 1%. The main finding…

      yogthosView on HNParent context

  8. Storm 2.0 1 comment

    • > this evidence has been around for ages Evidence of what? This is not a study of software development at all, but of prototyping using a program with no more than a few hundred lines. I'm talking about effects on software development. The issue of prototyping/tiny programs is a…

      pronView on HNParent context

  9. Correctness – A paradigm for sustainable software development 1 comment

    • That study: 1. Found a statistically significant, but a rather small effect (low single digit % IIRC) that cannot justify language/paradigm choice (choose FP for 1.5% fewer bugs!). If anything, it's evidence against a large effect. 2. Had most of even that small effect disappear…

      pronView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.