HackerPapers

arXiv:1802.08232cs.LG

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, Dawn Song

Abstract

This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization. In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence cs.CRCryptography and Security

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from May 2019 to Jan 2021. Found in 4 HN submissions from Feb 2018 to Oct 2018. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. DALL·E: Creating Images from Text 1 comment

    • Models can unpredictably memorize sensitive input data, so there can be a real copyright issue here, I think. https://arxiv.org/abs/1802.08232 Worse, sometimes the input data is illegal to distribute for other reasons than copyright.

      astrangeView on HNParent context

  2. A Plan Made to Shield Big Tobacco from Facts Is Now EPA Policy 1 comment

    • Point 1 might be less clear cut in some settings. Obviously, if you're just doing a simple regression, it's no big deal, but more powerful models can memorize and expose parts of their training data. See for example: https://arxiv.org/abs/1802.08232

      ImnimoView on HNParent context

  3. Training algorithms on copyrighted data not illegal: US Supreme Court 1 comment

    • We probably need a technical rather than legal solution for this. The problem is that generative algorithms are susceptible to accidental memorization so you can't guarantee that the output will be transformative. For example, play with https://talktotransformer.com to see how…

      fypView on HNParent context

  4. Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline 1 comment

    • It actually can be more pernicious than that: https://arxiv.org/abs/1802.08232 However note that the dataset used to train GPT-2 is about 20x smaller than C4. I'm not 100% sure how many times the training set was repeated over the course of training for GPT-2, but it was likely…

      craffelView on HNParent context

  5. Maryland research could improve the AI task of sensorimotor representation 1 comment

    • This is probably a "publishing in the press" translation issue, but the notion that neural networks have no memory is dead wrong. DNNs encode incredibly specific histories of training data in their weights, which is both a privacy/security risk [1] and a phenomena that is…

      evrydayhustlingView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.