HackerPapers

arXiv:2311.17035cs.LG

Scalable Extraction of Training Data from (Production) Language Models

Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, Katherine Lee

Abstract

This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training dataset. We show an adversary can extract gigabytes of training data from open-source language models like Pythia or GPT-Neo, semi-open models like LLaMA or Falcon, and closed models like ChatGPT. Existing techniques from the literature suffice to attack unaligned models; in order to attack the aligned ChatGPT, we develop a new divergence attack that causes the model to diverge from its chatbot-style generations and emit training data at a rate 150x higher than when behaving properly. Our methods show practical attacks can recover far more data than previously thought, and reveal that current alignment techniques do not eliminate memorization.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CLComputation and Language cs.CRCryptography and Security

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Dec 2023 to Nov 2025. Found in 4 HN submissions from Nov 2023 to Dec 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. The current state of the theory that GPL propagates to AI models 1 comment

    • > Like if there is no way to trace it back to the original material, does it make sense to regulate it? Training data extraction has seen some success, tracing should be possible for at least some of it https://arxiv.org/abs/2311.17035

      uhfraidView on HNParent context

  2. What GPT-OSS leaks about OpenAI's training data 1 comment

    • Yes. https://arxiv.org/abs/2403.06634 https://arxiv.org/abs/2311.17035 (I just have these ones off the top of my head because I'm a Nicholas Carlini fan and we interviewed him about these attacks.)

      tptacekView on HNParent context

  3. Beyond self-attention: How a small language model predicts the next token 1 comment

    • I had the exact same idea after seeing Google point out that you can[0] get ChatGPT to regurgitate verbatim training data by asking it to repeat the same word over and over again[1]. I'm glad to see someone else actually bring it to fruition. This, of course, brings two…

      kmeisthaxView on HNParent context

  4. Nightshade: An offensive tool for artists against AI art generators 1 comment

    • Commercial musicians do not? You sure about that? Entire legal firm empires have been built on the licensing, negotiations, and fees that make up the industry. I'm ain't talking about some dude on YouTube or Soundcloud. Few people care about some rando on Soundcloud. Those moles…

      snakeyjakeView on HNParent context

  5. Sorry, but a new prompt for GPT-4 is not a paper 1 comment

    • Sorry for any confusion, my comment was meant to refer to the linked paper: https://arxiv.org/abs/2311.17035 I think it’s valid work, but the original tweet seems to call a prompt based paper into question. At least enough to clarify where he would stand on an example like this.

      WhitneyLandView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.