HackerPapers

arXiv:2211.04325cs.LG

Will we run out of data? Limits of LLM scaling based on human-generated data

Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Marius Hobbhahn

Abstract

We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on current trends and estimate the total stock of public human text data. Our findings indicate that if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032, or slightly earlier if models are overtrained. We explore how progress in language modeling can continue when human-generated text datasets cannot be scaled any further. We argue that synthetic data generation, transfer learning from data-rich domains, and data efficiency improvements might support further progress.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence cs.CLComputation and Language cs.CVComputer Vision and Pattern Recognition cs.CYComputers and Society

On Hacker News

Linked by 4 identifiable commenters in 5 comments across 4 threads, from Dec 2022 to Sep 2026. Found in 6 HN submissions from Nov 2022 to Jun 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Nvidia wants to put a watchdog chip next to every AI agent 1 comment

    • Geoffrey Hinton thinks AI is conscious. I know he's extremely accomplished and all, but as it pertains to AI as it exists now we're miles apart (not to say that he's not brilliant, but by difference of opinion). I also don't want to rehash the whole what-is-consciousness…

      wavewranglerView on HNParent context

  2. Failing to Understand the Exponential, Again 2 comments

    • > Where are they going to get more data? This is a great question, but note that folks were freaking out about this a year or so ago and we seem to be doing fine. We seem to be making progress with some combination of synthetic training datasets on coding/math tasks, textbooks…

      theptipView on HNParent context

    • I don’t think this is true. See https://arxiv.org/html/2211.04325v2 for example.

      theptipView on HNParent context

  3. Dropbox: How to opt out of 3rd party AI partner access to your Dropbox 1 comment

    • I have no doubt that this "feature" is a backroom deal worth millions because OpenAI is running out of public internet data with which to improve its models. (See this paper from researchers at MIT and a few other schools which predicts that high-quality text training data will…

      hungrigekatzeView on HNParent context

  4. Drowning in AI Generated Garbage: the silent war we are fighting 1 comment

    • there's a paper that basically shows these LLMs are pretty much tapped out in terms of quality data to train on, not sure if there are any legit architecture improvements in the works to improve results quality but it seems brute force training with more data may be about over…

      ren_engineerView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.