HackerPapers

arXiv:2303.11156cs.CL

Can AI-Generated Text be Reliably Detected?

Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, Soheil Feizi

Abstract

Large Language Models (LLMs) perform impressively well in various applications. However, the potential for misuse of these models in activities such as plagiarism, generating fake news, and spamming has raised concern about their responsible use. Consequently, the reliable detection of AI-generated text has become a critical area of research. AI text detectors have shown to be effective under their specific settings. In this paper, we stress-test the robustness of these AI text detectors in the presence of an attacker. We introduce recursive paraphrasing attack to stress test a wide range of detection schemes, including the ones using the watermarking as well as neural network-based detectors, zero shot classifiers, and retrieval-based detectors. Our experiments conducted on passages, each approximately 300 tokens long, reveal the varying sensitivities of these detectors to our attacks. Our findings indicate that while our recursive paraphrasing method can significantly reduce detection rates, it only slightly degrades text quality in many cases, highlighting potential vulnerabilities in current detection systems in the presence of an attacker. Additionally, we investigate the susceptibility of watermarked LLMs to spoofing attacks aimed at misclassifying human-written text as AI-generated. We demonstrate that an attacker can infer hidden AI text signatures without white-box access to the detection method, potentially leading to reputational risks for LLM developers. Finally, we provide a theoretical framework connecting the AUROC of the best possible detector to the Total Variation distance between human and AI text distributions. This analysis offers insights into the fundamental challenges of reliable detection as language models continue to advance. Our code is publicly available at https://github.com/vinusankars/Reliability-of-AI-text-detectors.

arXiv comments
Published in Transactions on Machine Learning Research (TMLR)
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 5 identifiable commenters in 7 comments across 7 threads, from Mar 2023 to Aug 2026. Found in 1 HN submission from Mar 2023 to Mar 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Anthropic's 'watermark' text adulteration in Claude is a perversion of writing 1 comment

    • I'd encourage reading this paper, and literature on scaling laws in autoregressive models: https://arxiv.org/abs/2303.11156 Total variation distance has been measured to decrease as you scale a model, and that is the primary mechanism "watermarking" as discussed in the Anthropic…

      0x_rsView on HNParent context

  2. How Claude marks AI-generated content 1 comment

    • If we go by "fool's errand" as "needless or profitless endeavor", https://arxiv.org/abs/2303.11156 may already be a good enough answer to the paper you cited, so their work is already laid out for them. The green token idea is thoroughly attacked with much more effective…

      0x_rsView on HNParent context

  3. Show HN: AI Code Detector – detect AI-generated code with 95% accuracy 1 comment

    • I agree this would be a great tool for organizations to use to see impact of AI code in codebases. Engineers will probably be too lazy to modify the code enough to make it look less AI. You could probably enhance the robustness of your classifier with synthetic data like this. I…

      mendezaView on HNParent context

  4. OpenAI's comment to the NTIA on open model weights 1 comment

    • A great deal of harm is being caused by "AI detectors" today. There is an endless stream of stories about students who have their work flagged as "AI generated" by a detector when it wasn't - and then get failing grades, from teachers who don't understand that AI detectors are…

      simonwView on HNParent context

  5. OpenAI shuts down its AI Classifier due to poor accuracy 1 comment

    • https://arxiv.org/pdf/2301.10226.pdf Here's a decent paper on it. It covers private watermarking (you can't detect it exists without a key), resistance to modifications, etc. Essentially you wouldn't know it was there and you can't make simple modifications to fool it. OpenAI…

      thewataccountView on HNParent context

  6. Show HN: Pledge Your Human-Made Content 1 comment

    • Most likely not https://arxiv.org/abs/2303.11156 tl;dr - it's highly unreliable to detect them and because of the risk of false-positives it's not likely a good idea to rely on it. IMO the only way forward is a reputation based system. Also moving forward identity verification…

      thewataccountView on HNParent context

  7. Research shows we can only accurately identify AI writers about 50% of the time 1 comment

    • It matters because LLMs can tell plausible lies at incredible scale: marketing, propaganda, misinformation and disinformation, etc. Understanding whether content is AI generated would be a useful red flag, but we can't. Nor can supposed "AI detectors" do so with any reliability…

      VeenView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.