Training Compute-Optimal Large Language Models · HackerPapers discussions

Training Compute-Optimal Large Language Models — mentioned by namr2000

Tue, 16 Jun 2026 18:48:48 GMT

You don't have to train from scratch but you can. Distillation ends up being somewhere in the ballpark of 1000x faster to train [1]. It also comes with the huge advantage of not needing to create RLHF datasets, since you can just copy the behavior of the teacher model. This saves an enormous amount of labeling money at the cost of making the model behave similarly to the teacher. If you are training from scratch, you can look at LLM scaling laws to figure out roughly the compute budget you need to optimally train a model [2]. Based on [2] a 30B model needs something like 2e+23 FLOPS to train…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by andy99

Fri, 19 Dec 2025 10:48:28 GMT

The chinchilla paper says the “optimal” training data set size is about 20x the number of parameters (in tokens), see table 3: https://arxiv.org/pdf/2203.15556 Here they do 80B tokens for a 4B model.

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by adityaathalye

Wed, 06 Aug 2025 06:58:04 GMT

Earlier... Training Compute-Optimal Large Language Models (2022) https://arxiv.org/abs/2203.15556 Chinchilla Scaling: A replication attempt (2024) https://arxiv.org/abs/2404.10102

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by vonneumannstan

Fri, 09 May 2025 14:41:36 GMT

Pretty sure that since the Chinchilla paper this probably isn't the case. https://arxiv.org/pdf/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by versteegen

Sun, 17 Dec 2023 23:24:32 GMT

Because the training data/model size/compute tradeoff derived from that paper is highly suboptimal (too many parameters) compared to the ones from the later Deepmind scaling laws [1]. And then Meta researchers recommended using even smaller models, to trade-off training- and inference-time compute [2] (which I thought was pretty obvious if you care about more than just benchmarks). [1] https://arxiv.org/abs/2203.15556 Training Compute-Optimal Large Language Models [2] https://arxiv.org/abs/2302.13971 LLaMA: Open and Efficient Foundation Language Models

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by sa-code

Sat, 16 Dec 2023 08:19:58 GMT

I would say that the chinchilla paper is a prerequisite to all of the ones mentioned above https://arxiv.org/abs/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by huac

Thu, 19 Oct 2023 02:40:54 GMT

rebuttal: compute optimality matters https://arxiv.org/pdf/2203.15556.pdf

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by boomanaiden154

Sun, 17 Sep 2023 23:28:48 GMT

One of the biggest things that seems to be holding back ML in compilers right now is dataset size. This model was only trained on a gigabyte of source code, 30+% of that synthetic. Even on much simpler models, there have been massive performance gains by just throwing more data at them. Some experimentation with the original MLGO inlining model on a much bigger data corpus doubled the code-size wins. LLMs have also been shown to perform better they more data they are fed [1]. 1. https://arxiv.org/abs/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by naillo

Wed, 26 Jul 2023 19:21:51 GMT

They often reference this paper as the motivation for that https://arxiv.org/pdf/2203.15556.pdf I.e. training with 10x data and 10x longer can yield as good models as a gpt-3 model but with fewer weights (according to the paper) and the same principle applies in vision.

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by airgapstopgap

Tue, 18 Jul 2023 18:52:49 GMT

Do you realize that LLaMA-1 is just a very slightly smaller, comparably performing replication of Chinchilla [1], which DeepMind had completed a year prior to LLaMA's release? And has RLHF-ed into a suitable chatbot "Sparrow" [2] months earlier than ChatGPT was launched? To assume that Google doesn't have anything competitive with Meta is to say that their papers just so happen to contain recipes for Meta's models but they've arrived at those not through training and benchmarking but by divination and bullshitting. This, let us say, does not sound plausible. Then again, Microsoft uses LLaMA…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by gdiamos

Wed, 12 Jul 2023 20:01:07 GMT

This initial public alpha release of Lamini LLM fine tuning includes several advanced optimizations: 1. Chinchilla recipe smaller models pretrained on data increases inference speed 2. Instruction fine tuning training on a small high quality set of instructions unlocks the knowledge learned during foundation model training. 3. Latency constrained batching achieves high utilization under load during token generation 4. Containerized SLURM combines fast scheduling of SLURM with LLM containers 5. Mixed precision training uses matrix operations for training There are so many low hanging fruits in…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by thewataccount

Thu, 04 May 2023 01:45:30 GMT

It's from this paper - https://arxiv.org/abs/2203.15556 Like the sibling comment said - the proportion of training tokens to parameter size is very important, and there's a certain threshold needed to be met for it to be "fully trained". Usually you have a fixed amount of compute (budget/time essentially) - and in that case you want to pick the largest parameter count that you can fully train, and not the largest parameter count your hardware can support and then train that for less time. tl;dr - Small models with training over the chinchilla threshold can out perform large models that are…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by thewataccount

Thu, 04 May 2023 01:34:58 GMT

> It's a pretty well accepted fact now that bigger LLM = moar better without exceptions. That's not true, the amount of training is a MAJOR factor. See the Chinchilla paper - https://arxiv.org/abs/2203.15556 tl;dr - a "fully" trained small model can outperform a "undertrained" larger model. If you have a fixed amount of compute (budget), then you need to optimize for the largest model that you can fully train, and not simply up the parameter count. EDIT: Also you can't necessarily compare the parameter count across model architectures* This thing seems to outperform the finetuned 30B llama…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by ftxbro

Fri, 28 Apr 2023 04:51:42 GMT

Chinchilla scaling laws aren't necessarily used for the tradeoffs of training runs like this one. Here is the goal of Chinchilla scaling from https://arxiv.org/pdf/2203.15556.pdf : "In this work, we revisit the question: Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens? To answer this question, we model the final pre-training loss L(N, D) as a function of the number of model parameters N, and the number of training tokens, D. Since the computational budget C is a deterministic function FLOPs(N, D) of the number of seen training tokens and model…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by thewataccount

Wed, 19 Apr 2023 18:15:40 GMT

> But more parameters will always be better for more complex tasks. I think you should checkout this paper which discusses the relationship of performance and the ratio of training tokens to parameter count. https://arxiv.org/abs/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by ftxbro

Sat, 15 Apr 2023 03:38:28 GMT

The ones I had in mind were the newer 'Chinchilla' scaling laws (https://arxiv.org/pdf/2203.15556.pdf) vs. the older 'Kaplan' scaling laws (https://arxiv.org/pdf/2001.08361.pdf)

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by crakenzak

Mon, 10 Apr 2023 02:47:43 GMT

I agree with your general statement, but if one thing the recent DeepMind Chinchilla research paper showed us is that the size of the model (number of parameters) is much less of the determinant of model performance & quality than the amount of high-quality data (number of high-quality tokens). Their 70B chinchilla model significantly outperforms the 175B GPT3 model. Possibly where OpenAI has a leg up is their high-quality data sourcing & curating infrastructure and their RLHF mechanisms. Paper: https://arxiv.org/abs/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by versteegen

Wed, 29 Mar 2023 05:59:13 GMT

The degree to which transformers scale, as measured by loss (training objective) is known remarkably well! See [1]. There's a formula in there for the minimum loss you possibly achieve even with infinite compute and training data, and it's hardly less than Chinchilla's loss. The recent GPT-4 paper further reinforces that these scaling laws are real, because they predicted with high accuracy the loss the model would have based on data+compute used. Admittedly, the link between improvement in loss and improvement of capabilities may break down or be misleading. However, there's just not much…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by rcme

Wed, 29 Mar 2023 04:15:33 GMT

https://openreview.net/pdf?id=yzkSU5zdwD https://arxiv.org/pdf/2203.15556.pdf There were also some informal comparisons of GPT models with various parameter counts.

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by jacobn

Mon, 27 Mar 2023 21:32:13 GMT

Both matter, and returns fall off as you go further in one but not the other. The Chinchilla paper[0] established a simple scaling law for Large Language Models: model size and training tokens should grow at the same pace. Compute is then proportional to the product of model size & data quantity. That said, quality of data also matters a lot - OpenAI has had human labelers produce the data for their Reinforcement Learning from Human Feedback (RLHF), which has probably had a disproportionate impact on the success of ChatGPT compared to previous models, but that data is probably O(1%) of what…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by trc001

Sat, 25 Mar 2023 05:21:51 GMT

Well, there sort of is a linear relationship between the ammount of training data and the quality of the model [1] [1]: https://arxiv.org/abs/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by DavidSJ

Sat, 11 Mar 2023 17:24:09 GMT

GPT 3.5 likely differs from the original GPT 3 by more than instruction fine-tuning. For example, it was probably retrained under Chinchilla scaling laws [1], with a lot more data and maybe a somewhat smaller parameter count. There are many variants of GPT-3 and GPT-3.5, and based on the performance numbers in Meta’s paper, it looks like they’re comparing against the very first version of GPT-3 from 2020. [2] [1] https://arxiv.org/abs/2203.15556 [2] https://arxiv.org/abs/2005.14165

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by vishal0123

Sun, 26 Feb 2023 11:38:23 GMT

LLAMA made tradeoff for reducing parameter budget instead of training computation budget. This is better for inference computation budget. Optimal number of tokens for 7B parameters is around 140B tokens[0], and meta trained it for trillion tokens. [0]: https://arxiv.org/pdf/2203.15556.pdf

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by cubefox

Sat, 25 Feb 2023 14:27:21 GMT

This is the old scaling laws paper. The scaling laws in it turned out to be wrong and superseded by the Chinchilla DeepMind paper: https://arxiv.org/abs/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by lossolo

Sun, 19 Feb 2023 20:19:00 GMT

It's because BLOOM is undertrained, you can prune a lot of weights in BLOOM and it doesn't impact performance. Look at Chinchilla paper[1], 70B model outperforms 175B GPT-3 model. https://arxiv.org/abs/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by code_runner

Sat, 11 Feb 2023 12:47:46 GMT

I’ll do my best. Number of params is the number of weights. Basically the number of learnable variables. Number of tokens is how many tokens it saw during training. Vocab size is the number of distinct tokens. The relationship between params/tokens/compute power is something people have studied a good deal and how it affects model performance. https://arxiv.org/pdf/2203.15556.pdf

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by minimaxir

Mon, 06 Feb 2023 19:16:41 GMT

Given that Google/DeepMind is one of the few labs actually trying to make smaller models that are just as performant (e.g. Chinchilla: https://arxiv.org/pdf/2203.15556.pdf ), I'm optimistic.

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by sbierwagen

Tue, 27 Dec 2022 08:59:03 GMT

If the number of bits in the net is smaller than the number of bits in the training data, then training is always going to be a "lossy" operation, losing data in the strictest sense. So from that perspective, the source text is "more" "valuable". Training is a fairly mechanical operation-- you lay out the architecture, decide on an encoding/weighting scheme, then feed the training data through the net. Not a lot of secret sauce there. (As evidenced in the explosion in competing LLMs after the GPT papers were published) Going forward, the copyright status and the possible need to pay licensing…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by peaslock

Mon, 19 Dec 2022 08:01:48 GMT

> if you want improved performance, you still need more data Not true. See figure 2: https://arxiv.org/pdf/2203.15556.pdf#page=5 The loss decreases with greater model size at the same compute budget (i.e. stopping sooner regarding training data). Also some rehearsal/multi-epoch training improves the forgetting rate (thereby improving performance substantially), which hasn't been taken into account by Chinchilla et al. because they train <1 epoch. https://arxiv.org/abs/2205.12393

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by sinenomine

Sat, 17 Dec 2022 16:05:21 GMT

> When it comes to intelligence and cognition no one even knows where to begin Is it true, though? There is quite nice literature out there, surely John has read these papers during his bootstrap period: https://pubmed.ncbi.nlm.nih.gov/27683554/ https://arxiv.org/abs/2203.15556v1

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by naillo

Thu, 24 Nov 2022 12:29:02 GMT

They're motivating that choice via this paper: https://arxiv.org/pdf/2203.15556.pdf The paper shows that you can get better performance than gpt-3 with a much smaller model if you bump up the training time and training data like x4.

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by espadrine

Wed, 26 Oct 2022 12:08:12 GMT

BLOOM[0] is another 176B multilingual open-source model whose weights were published. However, running it locally requires 8 A6000 GPUs. The GPU cost alone is roughly $37K. Also, there is the concern of performance. The Chinchilla[1] paper indicates that most model sizes trained so far received too little token training (or were too big) to reach an optimal performance per energy expenditure. Furthermore, the InstructGPT[2] and more recently Flan[3] papers show that various finetuning techniques achieve significant gains, without which the result is a bit meh. The dark secret of GPT-3 is that…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by theptip

Thu, 22 Sep 2022 21:51:12 GMT

> But if you want to describe human activities in a text-to-video prompt (instead of using footage of real people as a guideline), and you’re expecting convincing and photoreal results that last more than 2-3 seconds, the system in question is going to need an extraordinary, almost Akashic knowledge about many more things than Stable Diffusion (or any other existing or planned deepfake system) knows anything about. > These include anatomy, psychology, basic anthropology, probability, gravity, kinematics, inverse kinematics, and physics, to name but a few. Worse, the system will need temporal…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — HN submission

Mon, 12 Sep 2022 18:30:11 GMT

Training Compute-Optimal Large Language Models

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by Calavar

Fri, 19 Aug 2022 23:51:40 GMT

One big issue is how exactly we'll continue to scale. Exponential growth is hard to maintain. An example: In the Chinchilla paper [1], the authors suggest that most big transformer models are undertrained, and that we will probably see diminishing returns in scaling up the size of networks if we don't also scale up the size of the datasets. They have a subanalysis where they extrapolate out how big datasets will need to be for larger models. If you believe it, then within two to three generations of big NLG models, we may need on the order of 100 trillion text tokens. But it's not clear if…

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — HN submission

Tue, 16 Aug 2022 19:17:21 GMT

Training Compute-Optimal Large Language Models

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by gardenfelder

Sun, 14 Aug 2022 20:07:49 GMT

https://arxiv.org/abs/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by lstamour

Mon, 11 Apr 2022 22:29:29 GMT

Tough to say. Technically https://arxiv.org/pdf/2203.15556.pdf has the same content, it just isn’t highlighted the same way.

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by timpetri

Sun, 10 Apr 2022 22:40:06 GMT

Question related to the Chinchilla paper[0], which says that optimal amount of training data for ~500B, 1T, and 10T param models are 11T, 21.2T, 216.2T tokens, respectively. The PaLM paper[1] says it made use of 700B tokens. How many tokens of training data have humans produced across the entire internet, all our written works, etc? Is there such a thing as a 216 trillion token set? [0] https://arxiv.org/abs/2203.15556 [1] https://arxiv.org/abs/2204.02311

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by _hark

Sun, 10 Apr 2022 19:51:46 GMT

DeepMind to OpenAI and everyone else[1]: > your hyperparameters are bad and you should feel bad It's amazing to me that such a big goof was missed by so many for so long. All these multimillion dollar language models and people just took the scaling laws at face value. [1]: https://arxiv.org/abs/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by mordymoop

Tue, 05 Apr 2022 02:31:27 GMT

Full paper: https://arxiv.org/abs/2203.15556

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — mentioned by lucidrains

Mon, 04 Apr 2022 22:36:36 GMT

revised scaling laws https://arxiv.org/abs/2203.15556 https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scalin...

Paper and discussion history. A mention is not an endorsement.

Training Compute-Optimal Large Language Models — HN submission

Sat, 02 Apr 2022 22:08:53 GMT

Training Compute-Optimal Large Language Models

8 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Training Compute-Optimal Large Language Models — HN submission

Wed, 30 Mar 2022 13:08:09 GMT

70B language model outperforms Gopher and GPT-3

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.