Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT · HackerPapers discussions

Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT — mentioned by sigtstp

Mon, 05 May 2025 08:17:15 GMT

I feel this makes some fundamental conceptual mistakes and is just riding the LLM wave. "Semantics" is literally behavior under execution. This is syntactical analysis by a stochastic language model. I know the NLP literature uses "semantics" to talk about representations but that is an assertion which is contested [1]. Coming back to testing, this implicitly relies on the strong assumption of the LLM correctly associating the code (syntax) with assertions of properties under execution (semantic properties). This is a very risky assumption considering, once again, these things are stochastic…

Paper and discussion history. A mention is not an endorsement.

Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT — mentioned by synthiq

Sat, 03 Jun 2023 16:39:45 GMT

For anyone interested in related research, I used https://mirrorthink.ai to find some background on the state-of-the-art. (disclaimer: this is AI generated, but grounded on contents of papers, with real references, so I'd say it is still constructive) The state-of-the-art in code generation has seen significant advancements with the deployment of large language models (LLMs) in various code authoring tools. One such example is the study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT [1], which evaluates the code quality of these AI-assisted code generation tools. The study reveals that…

Paper and discussion history. A mention is not an endorsement.