Scaling Laws and Interpretability of Learning from Repeated Data — mentioned by kouteiheika

Thu, 12 Jun 2025 03:32:50 GMT

In general this is of course an active area of research, but yes, you can do something that and people have done it successfully[1] by adding extra layers to an existing model and then continuing to train it. You have to be careful about the "same data" part though; ideally you want to train once on unique data[2] as excessive duplication can harm the performance of the model[3], although if you have limited data a couple of training epochs might be safe and actually improve the performance of the model[4]. [1] -- https://arxiv.org/abs/2312.15166 [2] -- https://arxiv.org/abs/1906.06669 [3]…

Paper and discussion history. A mention is not an endorsement.

Scaling Laws and Interpretability of Learning from Repeated Data — mentioned by p1esk

Wed, 03 Aug 2022 17:51:58 GMT

Oh, I actually meant to link to this paper: https://arxiv.org/pdf/2205.10487.pdf They make an argument that there might exist an unfortunate dup/unique data ratio in a dataset, where a model decides to memorize a frequently repeated chunk of data which is big enough to justify accuracy degradation happening for the rest of the data, but not too big to make memorization difficult (section 5.1). The degradation they show is substantial - almost as if going from 800M to 400M model.

Paper and discussion history. A mention is not an endorsement.

Scaling Laws and Interpretability of Learning from Repeated Data · HackerPapers discussions

Scaling Laws and Interpretability of Learning from Repeated Data — mentioned by kouteiheika

Scaling Laws and Interpretability of Learning from Repeated Data — mentioned by p1esk