HackerPapers

arXiv:2308.02312cs.SE

Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions

Samia Kabir, David N. Udo-Imeh, Bonan Kou, Tianyi Zhang

Abstract

Q&A platforms have been crucial for the online help-seeking behavior of programmers. However, the recent popularity of ChatGPT is altering this trend. Despite this popularity, no comprehensive study has been conducted to evaluate the characteristics of ChatGPT's answers to programming questions. To bridge the gap, we conducted the first in-depth analysis of ChatGPT answers to 517 programming questions on Stack Overflow and examined the correctness, consistency, comprehensiveness, and conciseness of ChatGPT answers. Furthermore, we conducted a large-scale linguistic analysis, as well as a user study, to understand the characteristics of ChatGPT answers from linguistic and human aspects. Our analysis shows that 52% of ChatGPT answers contain incorrect information and 77% are verbose. Nonetheless, our user study participants still preferred ChatGPT answers 35% of the time due to their comprehensiveness and well-articulated language style. However, they also overlooked the misinformation in the ChatGPT answers 39% of the time. This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.

arXiv comments
This paper has been conditionally accepted for the CHI Conference on Human Factors in Computing Systems (CHI'24). The new version of this paper has been modified with updated discussions to address more stakeholders
Journal reference
DOI
doi:10.1145/3613904.3642596
Subjects
cs.SESoftware Engineering cs.AIArtificial Intelligence

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 2 threads, from Aug 2023 to May 2025. Found in 8 HN submissions from Aug 2023 to May 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. Semantic unit testing: test code without executing it 1 comment

    • I feel this makes some fundamental conceptual mistakes and is just riding the LLM wave. "Semantics" is literally behavior under execution. This is syntactical analysis by a stochastic language model. I know the NLP literature uses "semantics" to talk about representations but…

      sigtstpView on HNParent context

  2. ChatGPT's odds of getting code questions correct are worse than a coin flip 1 comment

    • From the paper [1]: > For each of the 517 SO questions, the first two authors manually used the SO question's title, body, and tags to form one question prompt and fed that to the Chat Interface of ChatGPT. The generated answers by ChatGPT are then stored in CSV files. For the…

      tkgallyView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.