The Curse of Recursion: Training on Generated Data Makes Models Forget · HackerPapers discussions

The Curse of Recursion: Training on Generated Data Makes Models Forget — HN submission

Sun, 01 Dec 2024 05:21:45 GMT

The Curse of Recursion: Training on generated data makes models forget (2023)

122 points; 107 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by groby_b

Thu, 27 Jun 2024 21:23:22 GMT

You're confusing it with data poisoning. Model collapse itself is(was?) a fairly serious research topic: https://arxiv.org/abs/2305.17493 We've by now reached a "probably not inevitable" - https://arxiv.org/abs/2404.01413 argues there's a finite upper bound to error - but I'd also point out that that paper assumes training data cardinality increases with the number of training generations and is strictly accumulative. To a first order, that means you better have a pre-2022 dataset to get started, and have archived it well. but it's probably fair to say current SOTA is still more or less "it's…

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by vineyardlabs

Thu, 09 May 2024 18:21:16 GMT

https://arxiv.org/pdf/2305.17493 There's some cursory indication that in the long tail, training LLMs on LLM-generated data causes model collapse. Kind of like how if you photocopy a photocopy too many times the document becomes unreadable. This isn't really surprising though. Neural networks at large are a form of lossy compression. You can't do lossy compression on artifacts recovered from lossy compression too many times. The losses stack.

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by dangerwill

Tue, 16 Apr 2024 23:55:32 GMT

> Paradoxically, I think a lot of research is showing that synthetic training information can be just as good as the real stuff. Which studies show this? https://arxiv.org/abs/2305.17493 shows the exact opposite and my (layman's) understanding of statistics and epistemology lines up entirely with this finding. Like, how could this even theoretically work? In the best case scenario wouldn't training on synthetic training data make LLMs overconfident / overfit the data once faced with new (human) input to respond to?

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by almatabata

Tue, 05 Mar 2024 20:04:13 GMT

There is the threat of model collapse though as discussed in this paper: https://browse.arxiv.org/pdf/2305.17493v2.pdf? They discuss that if you train models with the output of other models you can get worse results. If we all start using code from those models how will that impact GPT-5?

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by gwern

Mon, 26 Feb 2024 18:47:57 GMT

> For better or worse, the quality of output LLMs can produce based on such publicly available data appears to be peaking. Recent work has documented that GPT, for example, has begun to exhibit declines in the quality of its output.3 It suggests no such thing, and the problems with citation 3 (https://arxiv.org/abs/2307.09009) have been extensively discussed. > It has been suggested this decline in performance is the expected result of a feedback loop, wherein data collected for training is increasingly contaminated by GPT itself, as users leverage the technology to produce and post content…

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by hprotagonist

Fri, 16 Feb 2024 15:16:49 GMT

> AI used to be weird. Now ‘sounds like a bot’ is just shorthand for boring. It's real, real hard to escape regression to the mean in autonomous systems. That gaussian's gonna get ya. [1] I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson, “The Curse of Recursion: Training on Generated Data Makes Models Forget,” May 2023, [Online]. Available: http://arxiv.org/abs/2305.17493

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by TimurSadekov

Fri, 12 Jan 2024 09:04:21 GMT

One of the world's leading experts on information security systems, Professor Ross Anderson of Cambridge University, in his article https://arxiv.org/abs/2305.17493v2 is already considering a scenario where most of the Internet will be a neural network-generated hallucination. But neither you, nor me, nor any expert or anyone at all will be able to distinguish between fake and real. As a result, neural networks will degenerate in the process of learning from their own hallucinations of the Internet, and humans will degenerate in the process of applying the degenerated neural networks. This…

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by TimurSadekov

Fri, 12 Jan 2024 08:51:53 GMT

Neural networks will degenerate in the process of learning from their own hallucinations, and humans will degenerate in the process of applying the degenerated neural networks. This process is called "neural network collapse". https://arxiv.org/abs/2305.17493v2 It can only be countered by a collective neural network of all the minds of humanity. For mutual validation and self-improvement of LLM and humans, we need the ability to match the knowledge of artificial intelligence with collective intelligence. Only the CyberPravda project is the practical solution to avoid the collapse of large…

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by kristopolous

Thu, 09 Nov 2023 06:25:28 GMT

maybe. A lot of modern clickwork is actually model training and there is a model-collapse phenomena (https://arxiv.org/abs/2305.17493) which means that it should be banned for such work. I bet a number of clever people on the platforms are already trying to instrument AI to do the work regardless - it's pretty close to "free money" if you can pull it off and not get caught and at a spigot size where there's no real serious consequences if you do.

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by kristopolous

Mon, 07 Aug 2023 12:29:26 GMT

multi-generational degradation is broadly called "model collapse" https://arxiv.org/pdf/2305.17493.pdf

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by YeGoblynQueenne

Tue, 01 Aug 2023 22:19:47 GMT

>> There's already been the studies that these models quickly break down when they're fed their own generated data as training data Not "these models" if you're referring to large language models pretrained and fine-tuned with Transformers. The only study I am aware of that explored the effect you discuss [1] had to make do with very different models as proxies: Variational Autoencoders, Mixture of Gaussians and a smaller LLM with 125 million parameters that hardly qualifies as one. It is not safe to extrapolate those results to really Large-LMs trained with Transformers, as it is not safe to…

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by concinds

Fri, 21 Jul 2023 20:01:55 GMT

Fundamentally untrue, and disheartening that it's the top comment. You can't use a model's output to train another model, it leads to complete gibberish (termed "model collapse"). https://arxiv.org/abs/2305.17493v2 And the Llama 2 license allows users to train derivative models, which is what people really care about. https://github.com/facebookresearch/llama/blob/main/LICENSE

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by martin_a

Sun, 09 Jul 2023 10:37:14 GMT

There are first studies on this topic: "THE CURSE OF RECURSION: TRAINING ON GENERATED DATA MAKES MODELS FORGET" https://arxiv.org/pdf/2305.17493.pdf

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by jerjerjer

Mon, 03 Jul 2023 08:42:45 GMT

Paper: https://arxiv.org/abs/2305.17493

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by throwaway4aday

Fri, 30 Jun 2023 19:28:04 GMT

That paper[0] mentions two defects, catastrophic forgetting which occurs during continual learning. GPT type models do not use this type of training and use supervised training instead which utilizes a curated corpus of text. The second is model collapse which is essentially just bad data making its way into the training set, once again a sufficiently curated corpus eliminates this problem. Microsoft has published a few papers about their research in using LLMs to train new LLM models. They have used training data consisting of essentially recorded chatGPT interactions which consist of the…

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by carbocation

Wed, 28 Jun 2023 21:17:58 GMT

Appears to be susceptible to model collapse[1], depending on how you do it. 1 = https://arxiv.org/abs/2305.17493v2

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by qayxc

Fri, 23 Jun 2023 22:01:27 GMT

Related: https://arxiv.org/abs/2305.17493v2 Basically, if this paper is right, 2021 was the last year usable training data could be pulled from the internet. It could be all downhill from hereon out, as the amount of AI generated content will soon overtake actual human generated and -curated data on the internet. So the party could be over soon ¯\_(ツ)_/¯

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — mentioned by spacemanspiff01

Wed, 14 Jun 2023 16:56:39 GMT

I believe that performance degrades, as the models end up training on data that is more homogenized and less diverse when compared to original data. At least according to this paper (I think, FYI I am not a expert) https://arxiv.org/abs/2305.17493

Paper and discussion history. A mention is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — HN submission

Tue, 13 Jun 2023 23:25:05 GMT

The Curse of Recursion: Training on Generated Data Makes Models Forget

170 points; 117 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Curse of Recursion: Training on Generated Data Makes Models Forget — HN submission

Sat, 03 Jun 2023 11:35:08 GMT

The Curse of Recursion: Training on Generated Data Makes Models Forget

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.