Scaling Data-Constrained Language Models · HackerPapers discussions

Scaling Data-Constrained Language Models — mentioned by wavewrangler

Wed, 30 Sep 2026 05:52:32 GMT

Geoffrey Hinton thinks AI is conscious. I know he's extremely accomplished and all, but as it pertains to AI as it exists now we're miles apart (not to say that he's not brilliant, but by difference of opinion). I also don't want to rehash the whole what-is-consciousness philosophical debate, and actually it's a separate question anyway, something doesn't have to be conscious to be dangerous. So I'll stick to the danger bit. Do you use AI much? Not a trick question. What is your usage like? Reason I ask is recall OpenAI hyping GPT-2 with the same language as they are these new models. Doom…

Paper and discussion history. A mention is not an endorsement.

Scaling Data-Constrained Language Models — mentioned by abrichr

Mon, 20 Jan 2025 13:51:38 GMT

I believe the paper being referenced is “Scaling Data-Constrained Language Models” (https://arxiv.org/abs/2305.16264). For correctness, you can use a solver to verify generated data.

Paper and discussion history. A mention is not an endorsement.

Scaling Data-Constrained Language Models — mentioned by ActivatedAI

Fri, 02 Aug 2024 18:10:35 GMT

There is some good published research about doing multiple passes over the training data, and how quickly learning saturates. The TL:DR is that diminishing returns kicks in after about 4 epochs. https://arxiv.org/abs/2305.16264

Paper and discussion history. A mention is not an endorsement.

Scaling Data-Constrained Language Models — mentioned by whimsicalism

Wed, 06 Sep 2023 02:07:04 GMT

Was this not sort of the clear implication of the fact that most LLMs are currently only being trained with one epoch? ie. if they are only being trained from one epoch, there is clear overfitting concerns just by doing even a second pass in the data. It does seem somewhat contrary to the findings of this paper [0] that found that old data was as good as new for at least 4 epochs. [0]: https://arxiv.org/abs/2305.16264

Paper and discussion history. A mention is not an endorsement.

Scaling Data-Constrained Language Models — mentioned by futurisold

Tue, 05 Sep 2023 09:18:17 GMT

You might want to give a read to "Scaling Data-Constrained Language Models" [1]. They basically generalized the Chinchilla scaling law by investigating behavior on multi-epoch runs. [1] https://arxiv.org/abs/2305.16264

Paper and discussion history. A mention is not an endorsement.

Scaling Data-Constrained Language Models — HN submission

Tue, 30 May 2023 22:01:02 GMT

Scaling Data-Constrained Language Models

50 points; 5 discussion replies on HN. Paper and source history. A submission is not an endorsement.