HackerPapers

arXiv:2303.15056cs.CL

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

Fabrizio Gilardi, Meysam Alizadeh, Maël Kubli

Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.

Abstract

Many NLP applications require manual data annotations for a variety of tasks, notably to train classifiers or evaluate the performance of unsupervised models. Depending on the size and degree of complexity, the tasks may be conducted by crowd-workers on platforms such as MTurk as well as trained annotators, such as research assistants. Using a sample of 2,382 tweets, we demonstrate that ChatGPT outperforms crowd-workers for several annotation tasks, including relevance, stance, topics, and frames detection. Specifically, the zero-shot accuracy of ChatGPT exceeds that of crowd-workers for four out of five tasks, while ChatGPT's intercoder agreement exceeds that of both crowd-workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk. These results show the potential of large language models to drastically increase the efficiency of text classification.

arXiv comments
Gilardi, Fabrizio, Meysam Alizadeh, and Maël Kubli. 2023. "ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks". Proceedings of the National Academy of Sciences 120(30): e2305016120
Journal reference
DOI
doi:10.1073/pnas.2305016120
Subjects
cs.CLComputation and Language cs.CYComputers and Society

On Hacker News

Linked by 1 identifiable commenter in 5 comments across 5 threads, from Jun 2023 to Sep 2023. Found in 1 HN submission from Mar 2023 to Mar 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Against LLM Maximalism 1 comment

    • LLMs are better at that though. Sure you may not require them but it certainly wouldn't be for a lack of accuracy. https://www.artisana.ai/articles/gpt-4-outperforms-elite-cro... https://arxiv.org/abs/2303.15056

      famouswafflesView on HNParent context

  2. Classifying customer messages with LLMs vs traditional ML 1 comment

    • Current State of the art (GPT-4) is mostly on par with experts and much better than crowdworkers. Might be overkill though. https://www.artisana.ai/articles/gpt-4-outperforms-elite-cro... 3.5 (what is used here) is better than crowd workers https://arxiv.org/abs/2303.15056

      famouswafflesView on HNParent context

  3. Mechanical Turk workers are using AI to automate being human 1 comment

    • Or just use the tools directly since they no longer underperform human actors. GPT-4 going toe to toe with the experts who set the benchmarks, way overperforming the crowdworkers https://www.artisana.ai/articles/gpt-4-outperforms-elite-cro... GPT-3.5 outperforming crowdworkers…

      famouswafflesView on HNParent context

  4. The idea maze for AI startups (2015) 1 comment

    • >should you just cut out the middle-model and be using ChatGPT as your classifier? Oh you certainly could. See here: GPT-3.5 outperforming elite crowdworkers on MTurk for Text annotation https://arxiv.org/abs/2303.15056 GPT-4 going toe to toe with expertrs (and significantly…

      famouswafflesView on HNParent context

  5. 33-46% of workers on MTurk used LLMs in a text production task 1 comment

    • There's the paper that shows ChatGPT outperforming MTurk Crowdworkers and another one showing GPT-4 going to to toe with experts (and blowing past crowdworkers) who set the benchmarks they were evaluating. https://arxiv.org/abs/2303.15056…

      famouswafflesView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.