HackerPapers

arXiv:2308.04477cs.SE

A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages

Alessio Buscemi

Abstract

Large Language Models (LLMs) are advanced Artificial Intelligence (AI) systems that have undergone extensive training using large datasets in order to understand and produce language that closely resembles that of humans. These models have reached a level of proficiency where they are capable of successfully completing university exams across several disciplines and generating functional code to handle novel problems. This research investigates the coding proficiency of ChatGPT 3.5, a LLM released by OpenAI in November 2022, which has gained significant recognition for its impressive text generating and code creation capabilities. The skill of the model in creating code snippets is evaluated across 10 various programming languages and 4 different software domains. Based on the findings derived from this research, major unexpected behaviors and limitations of the model have been identified. This study aims to identify potential areas for development and examine the ramifications of automated code generation on the evolution of programming languages and on the tech industry.

arXiv comments
Journal reference
DOI
Subjects
cs.SESoftware Engineering cs.AIArtificial Intelligence

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 2 threads, from Oct 2023 to Jul 2024. Found in 2 HN submissions from Nov 2023 to Dec 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. Julia for Economists (2022) 1 comment

    • There is a paper that claims that Julia performs the best in ChatGPT: https://arxiv.org/abs/2308.04477 Chris blog on that: https://www.stochasticlifestyle.com/chatgpt-performs-better-...

      minetest2048View on HNParent context

  2. Towards understanding sycophancy in language models 1 comment

    • I could swear I saw this the other day but I couldn’t find it in YOShInOn, here is an older paper which gets results like this for C/C++ but does a lot better with R, see Figure 1 https://arxiv.org/pdf/2308.04477.pdf The results of this kind of eval could be across the board…

      PaulHouleView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.