HackerPapers

arXiv:2402.14873cs.CL

Technical Report on the Pangram AI-Generated Text Classifier

Bradley Emi, Max Spero

Abstract

We present Pangram Text, a transformer-based neural network trained to distinguish text written by large language models from text written by humans. Pangram Text outperforms zero-shot methods such as DetectGPT as well as leading commercial AI detection tools with over 38 times lower error rates on a comprehensive benchmark comprised of 10 text domains (student writing, creative writing, scientific writing, books, encyclopedias, news, email, scientific papers, short-form Q&A) and 8 open- and closed-source large language models. We propose a training algorithm, hard negative mining with synthetic mirrors, that enables our classifier to achieve orders of magnitude lower false positive rates on high-data domains such as reviews. Finally, we show that Pangram Text is not biased against nonnative English speakers and generalizes to domains and models unseen during training.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence

On Hacker News

Linked by 8 identifiable commenters in 8 comments across 8 threads, from Oct 2024 to Jul 2026. Found in 2 HN submissions from Mar 2024 to Jul 2026. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments8 threads

  1. Detecting LLM-Generated Texts with “Classical” Machine Learning 1 comment

    • "Text is simply not information dense enough to be able to decode some arbitrary signal of provenance from it...it's a bad fiction to perpetuate that any of this is anything more than tarot card reading." Not true at all. Pangram is highly effective and has a very low false…

      WhitneyLandView on HNParent context

  2. The bottleneck might be the air in the room 1 comment

    • Their technical report (arxiv.org/abs/2402.14873) goes into it. They train a classifier on human text paired with AI-generated versions of the same documents. Dataset scale is comparable to LLM pretraining sets.

      salahadawiView on HNParent context

  3. Ask HN: How do systems (or people) detect when a text is written by an LLM 1 comment

    • Pangram is probably the best known example of a detector with low false positives, they have a research paper here: https://arxiv.org/pdf/2402.14873. They do have an API but not sure if you need to request access for it. For humans I think it just comes down to interacting with…

      moonuView on HNParent context

  4. Show HN: Showboat and Rodney, so agents can demo what they've built 1 comment

    • I'm not affiliated with it in any way, but you might want to check out the leading model https://arxiv.org/abs/2402.14873 and the mixed authorship detection https://www.pangram.com/blog/pangram-3-0-technical. LLM detection has come a long way since the first GPTZero. Human…

      usefulposterView on HNParent context

  5. Generative AI and Wikipedia editing: What we learned in 2025 1 comment

    • Human editors, I suspect, would pick up the "tells" of generated text, although as we know, there's a lot of false positives in that space. But it looks like Pangram is a text classifying NN trained using a technique where they get a human to write a body of text on a subject…

      EdwardDiegoView on HNParent context

  6. Amazon cuts 16k jobs 1 comment

    • AI detectors in general are unreliable, but there are a few made by serious researchers that have only 1-in-10000 false positive rate, e.g. https://arxiv.org/pdf/2402.14873 Having worked in a bigcorp, I've read my fair share of management-speak, and none of it sounds quite as…

      gamegoblinView on HNParent context

  7. Open-source Zig book 1 comment

    • I ran the introduction chapter through Pangram [1], which is one of the most reliable AI-generated text classifiers out there [2] (with a benchmarked accuracy of 99.85% over long-form text), and it gives high confidence for it having been AI-generated. It's also very intuitively…

      agucovaView on HNParent context

  8. AI Slop Is Flooding Medium 1 comment

    • We benchmark on pre-2023 datasets of O(10M) documents not in our training set. Other detectors seem to have between 1-3% false positive rate and ours is around 1 in 10,000 as of our latest model update. We do a lot of active learning + core set selection to keep FPR low and…

      maxsperoView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.