HackerPapers

arXiv:2501.15654cs.CL

People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text

Jenna Russell, Marzena Karpinska, Mohit Iyyer

Abstract

In this paper, we study how well humans can detect text generated by commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300 non-fiction English articles, label them as either human-written or AI-generated, and provide paragraph-length explanations for their decisions. Our experiments show that annotators who frequently use LLMs for writing tasks excel at detecting AI-generated text, even without any specialized training or feedback. In fact, the majority vote among five such "expert" annotators misclassifies only 1 of 300 articles, significantly outperforming most commercial and open-source detectors we evaluated even in the presence of evasion tactics like paraphrasing and humanization. Qualitative analysis of the experts' free-form explanations shows that while they rely heavily on specific lexical clues ('AI vocabulary'), they also pick up on more complex phenomena within the text (e.g., formality, originality, clarity) that are challenging to assess for automatic detectors. We release our annotated dataset and code to spur future research into both human and automated detection of AI-generated text.

arXiv comments
ACL 2025 33 pages
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 11 identifiable commenters in 13 comments across 13 threads, from Nov 2025 to Aug 2026. Found in 6 HN submissions from Jan 2025 to May 2026. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments13 threads

  1. I'm becoming AI-blind 1 comment

    • Most people do not realize when a personal message they receive was written by AI, study finds - https://theconversation.com/most-people-do-not-realize-when-... People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text…

      neomView on HNParent context

  2. Branchless Rust: Making a Filter 4x Faster by Removing an If 1 comment

    • Do you have evidence Pangram is unreliable? There are independent evaluations [1, 2] showing it works, and it's getting used more and more scientific papers. Have you used it or evaluated it yourself? What do you think these other evaluators are doing or getting wrong? 1…

      Seattle3503View on HNParent context

  3. Where .env Went Wrong 1 comment

    • This has been pretty well studied. Pangram is about as good at an expert human with a 98% detection rate and <2% false positive, for flagging AI generated text. https://arxiv.org/pdf/2501.15654

      chisView on HNParent context

  4. Ask HN: Add flag for AI-generated articles 1 comment

    • On the humans (vs. pangram) ability to reliably spot LLM generated content, there was an independent study: https://arxiv.org/html/2501.15654v1

      Chu4eenoView on HNParent context

  5. Looking Ahead to Postgres 19 1 comment

    • It has extremely low false positive rate in various tests (e. g. https://arxiv.org/html/2501.15654v1), I've never seen it mark something I knew was human written as AI.

      Chu4eenoView on HNParent context

  6. "No, I swear I wrote this." 1 comment

    • It's also confirmed in third-party studies like this https://arxiv.org/html/2501.15654v1 (which also confirms that humans that have experience with LLMs are very good at spotting them).

      Chu4eenoView on HNParent context

  7. I'm Tired of Talking to AI 1 comment

    • Humans writing more like LLMs, just like new LLMs write more like humans, it's all coalescing into one. I've copied-pasted comments I made on HN from like 2020 and had it tell me it's "100% AI". I've seen examples where the services claim "100% AI" because there was no normal…

      embedding-shapeView on HNParent context

  8. At least 25 Flock cameras have been destroyed in five states since April 2025 1 comment

    • Rather obviously they're choosing the one that makes them look best. Another they link to¹ shows 98% for example. Much more importantly, 9/10 dentists agree it's the best. 1: https://arxiv.org/pdf/2501.15654, linked from² https://www.pangram.com/blog/third-party-pangram-evals…

      GroxxView on HNParent context

  9. Fisker went bankrupt and owners built an open source car company from the ashes 1 comment

    • This reads very AI. Pangram [0] agrees [1]. [0] Not perfect, but I think as good evidence as any: https://arxiv.org/pdf/2501.15654 [1] https://www.pangram.com/history/44cd07d3-ba94-4331-8c7f-a626...

      kaonwarbView on HNParent context

  10. We found an undocumented bug in the Apollo 11 guidance computer code 1 comment

    • There is research showing the contrary that is far more convincing: > Our experiments show that annotators who frequently use LLMs for writing tasks excel at detecting AI-generated text, even without any specialized training or feedback. In fact, the majority vote among five…

      nothinkjustaiView on HNParent context

  11. Ask HN: How do systems (or people) detect when a text is written by an LLM 1 comment

    • Pangram has time after time been shown as the only detector that mostly works. And that paper is pretty old now! There are recent papers from academics independently bench-marking and studying detectors e.g. https://arxiv.org/abs/2501.15654

      spindump8930View on HNParent context

  12. “You should never build a CMS” 1 comment

    • I don't think people should be obligated to spend time and effort justifying their reasoning on this. Firstly it's highly asymmetrical; you can generate AI content with little effort, whereas composing a detailed analysis requires a lot more work. It's also not easily…

      samdoesnothingView on HNParent context

  13. When your hash becomes a string: Hunting Ruby's million-to-one memory bug 1 comment

    • Human experts can reliably detect some kinds of long-form, AI-generated text using exactly the same sorts of cues I've outlined: https://arxiv.org/html/2501.15654v1. You may take issue with the quality of the paper, but there have been very few studies like this and this one…

      Jweb_GuruView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.