Where Is Self-admitted Code Generated by Large Language Models on GitHub?

Abstract

The increasing use of Large Language Models (LLMs) in software development has garnered significant attention from researchers evaluating the capabilities and limitations of LLMs for code generation. However, much of the research focuses on controlled datasets such as HumanEval, which do not adequately capture the characteristics of LLM-generated code in real-world development scenarios. To address this gap, our study investigates self-admitted code generated by LLMs on GitHub, specifically focusing on instances where developers in projects with over five stars acknowledge the use of LLMs to generate code through code comments. Our findings reveal several key insights: (1) ChatGPT and Copilot dominate code generation, with minimal contributions from other LLMs. (2) Projects containing ChatGPT/Copilot-generated code appears in small/medium-sized projects led by small teams, which are continuously evolving. (3) ChatGPT/Copilot-generated code generally is a minor project portion, primarily generating short/moderate-length, low-complexity snippets (e.g., algorithms and data structures code; text processing code). (4) ChatGPT/Copilot-generated code generally undergoes minimal modifications, with bug-related changes ranging from 4% to 12%. (5) Most code comments only state LLM use, while few include details like prompts, human edits, or code testing status. Based on these findings, we discuss the implications for researchers and practitioners.