HackerPapers

arXiv:2305.16264cs.CL

Scaling Data-Constrained Language Models

Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, Colin Raffel

Abstract

The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.

arXiv comments
50 pages (9 main), 39 figures, 15 tables
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 5 identifiable commenters in 5 comments across 5 threads, from Sep 2023 to Sep 2026. Found in 1 HN submission from May 2023 to May 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Nvidia wants to put a watchdog chip next to every AI agent 1 comment

    • Geoffrey Hinton thinks AI is conscious. I know he's extremely accomplished and all, but as it pertains to AI as it exists now we're miles apart (not to say that he's not brilliant, but by difference of opinion). I also don't want to rehash the whole what-is-consciousness…

      wavewranglerView on HNParent context

  2. FrontierMath was funded by OpenAI 1 comment

    • I believe the paper being referenced is “Scaling Data-Constrained Language Models” (https://arxiv.org/abs/2305.16264). For correctness, you can use a solver to verify generated data.

      abrichrView on HNParent context

  3. IsoFLOP curves of large language models are flat 1 comment

    • There is some good published research about doing multiple passes over the training data, and how quickly learning saturates. The TL:DR is that diminishing returns kicks in after about 4 epochs. https://arxiv.org/abs/2305.16264

      ActivatedAIView on HNParent context

  4. Can LLMs learn from a single example? 1 comment

    • Was this not sort of the clear implication of the fact that most LLMs are currently only being trained with one epoch? ie. if they are only being trained from one epoch, there is clear overfitting concerns just by doing even a second pass in the data. It does seem somewhat…

      whimsicalismView on HNParent context

  5. Chinchilla’s death 1 comment

    • You might want to give a read to "Scaling Data-Constrained Language Models" [1]. They basically generalized the Chinchilla scaling law by investigating behavior on multi-epoch runs. [1] https://arxiv.org/abs/2305.16264

      futurisoldView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.