HackerPapers

arXiv:2202.07646cs.LG

Quantifying Memorization Across Neural Language Models

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, Chiyuan Zhang

Abstract

Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated easy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others). We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data. Memorization significantly grows as we increase (1) the capacity of a model, (2) the number of times an example has been duplicated, and (3) the number of tokens of context used to prompt the model. Surprisingly, we find the situation becomes more complicated when generalizing these results across model families. On the whole, we find that memorization in LMs is more prevalent than previously believed and will likely get worse as models continues to scale, at least without active mitigations.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CLComputation and Language

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 6 threads, from Aug 2022 to Jul 2026. Found in 1 HN submission from Mar 2023 to Mar 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. The Kimi K3 Moment 1 comment

    • I am not arguing that Kimi did not distill Opus or Sonnet (which they almost certainly have, as opposed to Fable or Sol). I am arguing that even if they did not distill, the model could still identify itself as Opus or Sonnet. That being said, I'd like to point out a few things…

      johndoughView on HNParent context

  2. Migrating away from Rust 1 comment

    • I'm not making an argument from grievance about my own code being plagiarized. I actually don't care if my own code is used without even the attribution required by the permissive licenses it's released under; I just want it to be used. I do also write proprietary code, but…

      mwcampbellView on HNParent context

  3. Judge dismisses majority of GitHub Copilot copyright claims 1 comment

    • "Can anyone find the filing?" https://arxiv.org/pdf/2202.07646 Personally, I would not rely on blogs like "developer-tech.com" for unbiased information about "AI" litigation. I would read the order and draw own conclusions.^1 (Note the plaintiffs are attempting to file an…

      1vuio0pswjnm7View on HNParent context

  4. We can't all use AI. Someone has to generate the training data 1 comment

    • > They just show it to the AI, it doesn't store it - just data about it. The same as you when you look at something, be it a painting or a chair. In general for machine learning, yes, but deep learning models with large numbers of parameters such as transformers and stable…

      versteegenView on HNParent context

  5. Ask HN: What can I personally do to fight AGAINST the Copilot class action? 1 comment

    • Deduplicating Training Data Makes Language Models Better https://arxiv.org/pdf/2107.06499 Quantifying Memorization Across Neural Language Models https://arxiv.org/pdf/2202.07646.pdf

      visargaView on HNParent context

  6. Exploring 12M of the 2.3B images used to train Stable Diffusion 1 comment

    • You are probably very aware of it, but just to highlight the importance of this for people who aren't aware: data duplication degrades the training and makes memorization (and therefore plagiarism, in the technical sense) more likely. For language models, this includes…

      ijkView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.