HackerPapers

arXiv:2301.13188cs.CR

Extracting Training Data from Diffusion Models

Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, Eric Wallace

Abstract

Image diffusion models such as DALL-E 2, Imagen, and Stable Diffusion have attracted significant attention due to their ability to generate high-quality synthetic images. In this work, we show that diffusion models memorize individual images from their training data and emit them at generation time. With a generate-and-filter pipeline, we extract over a thousand training examples from state-of-the-art models, ranging from photographs of individual people to trademarked company logos. We also train hundreds of diffusion models in various settings to analyze how different modeling and data decisions affect privacy. Overall, our results show that diffusion models are much less private than prior generative models such as GANs, and that mitigating these vulnerabilities may require new advances in privacy-preserving training.

arXiv comments
Journal reference
DOI
Subjects
cs.CRCryptography and Security cs.CVComputer Vision and Pattern Recognition cs.LGMachine Learning

On Hacker News

Linked by 17 identifiable commenters in 18 comments across 17 threads, from Feb 2023 to Jul 2025. Found in 2 HN submissions from Jan 2023 to Jan 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments17 threads

  1. Meta says it won't sign Europe AI agreement 1 comment

    • Then they should easily fall within the regulation section posted earlier. If you cannot see the difference between BitTorrent and Ai models, then it's probably not worth engaging with you. But Ai model have been shown to reproduce the training data…

      zizeeView on HNParent context

  2. Stable Diffusion 3 1 comment

    • If 1 in 1,000 generations will randomly produce memorized CSAM that slipped into the training set then yeah, it's pretty damn unsafe to use. Producing memorized images has precedent[0]. Is it unlikely? Sure, but worth validating. [0] https://arxiv.org/abs/2301.13188

      causalView on HNParent context

  3. Nightshade: An offensive tool for artists against AI art generators 1 comment

    • Commercial musicians do not? You sure about that? Entire legal firm empires have been built on the licensing, negotiations, and fees that make up the industry. I'm ain't talking about some dude on YouTube or Soundcloud. Few people care about some rando on Soundcloud. Those moles…

      snakeyjakeView on HNParent context

  4. OpenAI says it's "impossible" to create AI models without copyrighted material 2 comments

    • >Sites have been pushing towards "to continue reading, download our app" or "you've reached your free article limit, subscribe for just $X" for a long time, because it's profitable. I don't think there's sense in retroactively placing the blame for this behavior on crawlers And…

      johnnyanmacView on HNParent context

    • > I don't think it's a coincidence that the paywalling of more and more sites came as GDPR and other data privacy regulations started to pop up Data protection regulations being the cause of paywalls seems a different theory than what you were previously suggesting. > all those…

      UkvView on HNParent context

  5. Let's Not Flip Sides on IP Maximalism Because of AI 1 comment

    • https://www.technologyreview.com/2023/02/03/1067786/ai-model... The approach I've seen is to prompt for people with unusual names, they're often be only a single source image in the input data set that gets reproduced by the AI. I've seen examples with the AI "generated" images…

      MarazanView on HNParent context

  6. Judge pares down artists' AI copyright lawsuit against Midjourney, Stability AI 1 comment

    • > Compression implies the input can be reconstructed from the output (lossy or not), in the case of these ml models the input is the training data and the output is the model. You can't reconstruct even a fraction of that training data using the model alone therefore it is not…

      raincoleView on HNParent context

  7. Valve is not willing to publish games with AI generated content anymore? 1 comment

    • https://arxiv.org/pdf/2301.13188.pdf Stable Diffusion spits out slightly blurrier versions of the pictures in its training set.

      mehlmaoView on HNParent context

  8. GitHub Copilot emits GPL code 1 comment

    • Most generative AI actually does have significant problems with the model copying the data into itself. Not literally - there isn't a bunch of model parameters that line up to the exact PNG bitstream of particular images. But courts wouldn't care as long as the model outputs…

      kmeisthaxView on HNParent context

  9. Record labels panic as A.I. generated song becomes ‘hit single’ 1 comment

    • Most current "AI" models essentially compress and then store the information, often to the point where training data can be extracted [1][2] [1]: https://arxiv.org/abs/2012.07805 [2]: https://arxiv.org/abs/2301.13188

      vitehozonageView on HNParent context

  10. Universal Music Asks Streaming Services to Block AI Access to Its Songs 1 comment
  11. Stable Diffusion copyright lawsuits could be a legal earthquake for AI 1 comment

    • My mistake! I mis-remembered the original Google paper[0], which stated only that the image was in the Stable Diffusion training set. The complete set of "matches" they got were interesting, even while the number was very low. 0. https://arxiv.org/abs/2301.13188

      pwinnskiView on HNParent context

  12. Launch HN: Flower (YC W23) – Train AI models on distributed or sensitive data 1 comment

    • Every agent training the model on their proprietary data has to have access to the model form in some way (otherwise how would they train it?) For this reason, one must assume that the model form is known to the adversary. With this, the question becomes: is it possible to…

      aix1View on HNParent context

  13. Stable Attribution 1 comment

    • I agree with you that they have the capacity to remember an image - but they're not compressing them. That's a fundamentally different thing. The argument being made by that class action lawsuit is that "this thing can reproduce image X so it's a compression algorithm and…

      pksebbenView on HNParent context

  14. Getty Images v. Stability AI – Complaint 1 comment

    • IANAL either, but there are attacks you can use to extract the original training data from the models: https://arxiv.org/abs/2301.13188. I’d guess these attacks will get better with time. So in some ways, you can argue that Stability is also directly redistributing the original…

      nimithrynView on HNParent context

  15. An IP attorney’s reading of the Stable Diffusion class action lawsuit 1 comment

    • Sometimes, you're just wrong. "Image diffusion models such as DALL-E 2, Imagen, and Stable Diffusion have attracted significant attention due to their ability to generate high-quality synthetic images. In this work, we show that diffusion models memorize individual images from…

      wokwokwokView on HNParent context

  16. Extracting training data from diffusion models 1 comment

    • The tweet/paper co-author posted the paper (https://arxiv.org/abs/2301.13188) yesterday on HN (https://news.ycombinator.com/item?id=34596187) and ironically the top comment there is referencing to this exact tweet thread (which was posted yesterday as well). Evidence for the…

      babel_View on HNParent context

  17. LLM discourse needs more nuance 1 comment

    • Even lawyers aren't sure, the arguments can go either way: https://youtu.be/G08hY8dSrUY Also note some new research: https://arxiv.org/abs/2301.13188

      ben_wView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.