HackerPapers

arXiv:2203.15556cs.CL

Training Compute-Optimal Large Language Models

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre

Abstract

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 35 identifiable commenters in 40 comments across 37 threads, from Apr 2022 to Jun 2026. Found in 4 HN submissions from Mar 2022 to Sep 2022. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments37 threads

  1. Cohere's First Model for Developers 1 comment

    • You don't have to train from scratch but you can. Distillation ends up being somewhere in the ballpark of 1000x faster to train [1]. It also comes with the huge advantage of not needing to create RLHF datasets, since you can just copy the behavior of the teacher model. This…

      namr2000View on HNParent context

  2. History LLMs: Models trained exclusively on pre-1913 texts 1 comment

    • The chinchilla paper says the “optimal” training data set size is about 20x the number of parameters (in tokens), see table 3: https://arxiv.org/pdf/2203.15556 Here they do 80B tokens for a 4B model.

      andy99View on HNParent context

  3. The wall confronting large language models 1 comment

    • Earlier... Training Compute-Optimal Large Language Models (2022) https://arxiv.org/abs/2203.15556 Chinchilla Scaling: A replication attempt (2024) https://arxiv.org/abs/2404.10102

      adityaathalyeView on HNParent context

  4. Why do LLMs have emergent properties? 1 comment
  5. Do large language models need all those layers? 1 comment

    • Because the training data/model size/compute tradeoff derived from that paper is highly suboptimal (too many parameters) compared to the ones from the later Deepmind scaling laws [1]. And then Meta researchers recommended using even smaller models, to trade-off training- and…

      versteegenView on HNParent context

  6. Ask HN: AI/ML papers to catch up with current state of AI? 1 comment

    • I would say that the chinchilla paper is a prerequisite to all of the ones mentioned above https://arxiv.org/abs/2203.15556

      sa-codeView on HNParent context

  7. Consortium launched to build the largest open LLM 1 comment
  8. Large Language Models for Compiler Optimization 1 comment

    • One of the biggest things that seems to be holding back ML in compilers right now is dataset size. This model was only trained on a gigabyte of source code, 30+% of that synthetic. Even on much simpler models, there have been massive performance gains by just throwing more data…

      boomanaiden154View on HNParent context

  9. Stable Diffusion XL 1.0 1 comment

    • They often reference this paper as the motivation for that https://arxiv.org/pdf/2203.15556.pdf I.e. training with 10x data and 10x longer can yield as good models as a gpt-3 model but with fewer weights (according to the paper) and the same principle applies in vision.

      nailloView on HNParent context

  10. Llama 2 1 comment

    • Do you realize that LLaMA-1 is just a very slightly smaller, comparably performing replication of Chinchilla [1], which DeepMind had completed a year prior to LLaMA's release? And has RLHF-ed into a suitable chatbot "Sparrow" [2] months earlier than ChatGPT was launched? To…

      airgapstopgapView on HNParent context

  11. Free and Fast LLM Finetuning 1 comment

    • This initial public alpha release of Lamini LLM fine tuning includes several advanced optimizations: 1. Chinchilla recipe smaller models pretrained on data increases inference speed 2. Instruction fine tuning training on a small high quality set of instructions unlocks the…

      gdiamosView on HNParent context

  12. SparseGPT: Language Models Can Be Accurately Pruned in One-Shot 1 comment

    • It's from this paper - https://arxiv.org/abs/2203.15556 Like the sibling comment said - the proportion of training tokens to parameter size is very important, and there's a certain threshold needed to be met for it to be "fully trained". Usually you have a fixed amount of…

      thewataccountView on HNParent context

  13. Replit's new Code LLM: Open Source, 77% smaller than Codex, trained in 1 week 1 comment

    • > It's a pretty well accepted fact now that bigger LLM = moar better without exceptions. That's not true, the amount of training is a MAJOR factor. See the Chinchilla paper - https://arxiv.org/abs/2203.15556 tl;dr - a "fully" trained small model can outperform a "undertrained"…

      thewataccountView on HNParent context

  14. Llama 1.3B Trained on 200B Tokens for Commercial Use 1 comment

    • Chinchilla scaling laws aren't necessarily used for the tradeoffs of training runs like this one. Here is the goal of Chinchilla scaling from https://arxiv.org/pdf/2203.15556.pdf : "In this work, we revisit the question: Given a fixed FLOPs budget, how should one trade-off model…

      ftxbroView on HNParent context

  15. StableLM: A new open-source language model 1 comment

    • > But more parameters will always be better for more complex tasks. I think you should checkout this paper which discusses the relationship of performance and the ratio of training tokens to parameter count. https://arxiv.org/abs/2203.15556

      thewataccountView on HNParent context

  16. Sam Altman: OpenAI is not training GPT-5 and "won't for some time" 1 comment

    • The ones I had in mind were the newer 'Chinchilla' scaling laws (https://arxiv.org/pdf/2203.15556.pdf) vs. the older 'Kaplan' scaling laws (https://arxiv.org/pdf/2001.08361.pdf)

      ftxbroView on HNParent context

  17. The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT 1 comment

    • I agree with your general statement, but if one thing the recent DeepMind Chinchilla research paper showed us is that the size of the model (number of parameters) is much less of the determinant of model performance & quality than the amount of high-quality data (number of…

      crakenzakView on HNParent context

  18. Pause Giant AI Experiments: An Open Letter 1 comment

    • The degree to which transformers scale, as measured by loss (training objective) is known remarkably well! See [1]. There's a formula in there for the minimum loss you possibly achieve even with infinite compute and training data, and it's hardly less than Chinchilla's loss. The…

      versteegenView on HNParent context

  19. What we still don’t know about how A.I. is trained 1 comment

    • https://openreview.net/pdf?id=yzkSU5zdwD https://arxiv.org/pdf/2203.15556.pdf There were also some informal comparisons of GPT models with various parameter counts.

      rcmeView on HNParent context

  20. The Prospect of an AI Winter 1 comment

    • Both matter, and returns fall off as you go further in one but not the other. The Chinchilla paper[0] established a simple scaling law for Large Language Models: model size and training tokens should grow at the same pace. Compute is then proportional to the product of model…

      jacobnView on HNParent context

  21. GPT-4 performs significantly worse on coding problems not in its training data 1 comment

    • Well, there sort of is a linear relationship between the ammount of training data and the quality of the model [1] [1]: https://arxiv.org/abs/2203.15556

      trc001View on HNParent context

  22. Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support 1 comment

    • GPT 3.5 likely differs from the original GPT 3 by more than instruction fine-tuning. For example, it was probably retrained under Chinchilla scaling laws [1], with a lot more data and maybe a somewhat smaller parameter count. There are many variants of GPT-3 and GPT-3.5, and…

      DavidSJView on HNParent context

  23. LLaMA: A foundational, 65B-parameter large language model 2 comments

    • This is the old scaling laws paper. The scaling laws in it turned out to be wrong and superseded by the Chinchilla DeepMind paper: https://arxiv.org/abs/2203.15556

      cubefoxView on HNParent context

    • LLAMA made tradeoff for reducing parameter budget instead of training computation budget. This is better for inference computation budget. Optimal number of tokens for 7B parameters is around 140B tokens[0], and meta trained it for trillion tokens. [0]…

      vishal0123View on HNParent context

  24. Open source solution replicates ChatGPT training process 1 comment

    • It's because BLOOM is undertrained, you can prune a lot of weights in BLOOM and it doesn't impact performance. Look at Chinchilla paper[1], 70B model outperforms 175B GPT-3 model. https://arxiv.org/abs/2203.15556

      lossoloView on HNParent context

  25. A GPT in 60 Lines of NumPy 1 comment

    • I’ll do my best. Number of params is the number of weights. Basically the number of learnable variables. Number of tokens is how many tokens it saw during training. Vocab size is the number of distinct tokens. The relationship between params/tokens/compute power is something…

      code_runnerView on HNParent context

  26. Bard and new AI features in Search 1 comment

    • Given that Google/DeepMind is one of the few labs actually trying to make smaller models that are just as performant (e.g. Chinchilla: https://arxiv.org/pdf/2203.15556.pdf ), I'm optimistic.

      minimaxirView on HNParent context

  27. Ask HN: What's the value ratio of GPT3's LLM vs. its training data? 1 comment

    • If the number of bits in the net is smaller than the number of bits in the training data, then training is always going to be a "lossy" operation, losing data in the strictest sense. So from that perspective, the source text is "more" "valuable". Training is a fairly mechanical…

      sbierwagenView on HNParent context

  28. Copilot Internals 1 comment

    • > if you want improved performance, you still need more data Not true. See figure 2: https://arxiv.org/pdf/2203.15556.pdf#page=5 The loss decreases with greater model size at the same compute budget (i.e. stopping sooner regarding training data). Also some rehearsal/multi-epoch…

      peaslockView on HNParent context

  29. John Carmack Leaves Meta 1 comment

    • > When it comes to intelligence and cognition no one even knows where to begin Is it true, though? There is quite nice literature out there, surely John has read these papers during his bootstrap period: https://pubmed.ncbi.nlm.nih.gov/27683554/ https://arxiv.org/abs/2203.15556v1

      sinenomineView on HNParent context

  30. Stable Diffusion 2.0 1 comment

    • They're motivating that choice via this paper: https://arxiv.org/pdf/2203.15556.pdf The paper shows that you can get better performance than gpt-3 with a much smaller model if you bump up the training time and training data like x4.

      nailloView on HNParent context

  31. GPT-3 can create both sides of an Interactive Fiction transcript 1 comment

    • BLOOM[0] is another 176B multilingual open-source model whose weights were published. However, running it locally requires 8 A6000 GPUs. The GPU cost alone is roughly $37K. Also, there is the concern of performance. The Chinchilla[1] paper indicates that most model sizes trained…

      espadrineView on HNParent context

  32. The road to realistic full-body deepfakes 1 comment

    • > But if you want to describe human activities in a text-to-video prompt (instead of using footage of real people as a guideline), and you’re expecting convincing and photoreal results that last more than 2-3 seconds, the system in question is going to need an extraordinary…

      theptipView on HNParent context

  33. John Carmack's new AGI company, Keen Technologies, has raised a $20M round 1 comment

    • One big issue is how exactly we'll continue to scale. Exponential growth is hard to maintain. An example: In the Chinchilla paper [1], the authors suggest that most big transformer models are undertrained, and that we will probably see diminishing returns in scaling up the size…

      CalavarView on HNParent context

  34. Language Models Can Teach Themselves to Program Better 1 comment
  35. DeepMind’s New Language Model, Chinchilla 1 comment

    • Tough to say. Technically https://arxiv.org/pdf/2203.15556.pdf has the same content, it just isn’t highlighted the same way.

      lstamourView on HNParent context

  36. Estimating PaLM's Training Cost 2 comments

    • DeepMind to OpenAI and everyone else[1]: > your hyperparameters are bad and you should feel bad It's amazing to me that such a big goof was missed by so many for so long. All these multimillion dollar language models and people just took the scaling laws at face value. [1]…

      _harkView on HNParent context

    • Question related to the Chinchilla paper[0], which says that optimal amount of training data for ~500B, 1T, and 10T param models are 11T, 21.2T, 216.2T tokens, respectively. The PaLM paper[1] says it made use of 700B tokens. How many tokens of training data have humans produced…

      timpetriView on HNParent context

  37. Pathways Language Model (PaLM): Scaling to 540B parameters 2 comments

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.