Scaling Laws for Neural Language Models

Pretty much. See “Scaling Laws for Neural Language Models”, a must read IMO https://arxiv.org/html/2001.08361v1 Compute, data, and parameter count are the three logarithmic values that correspond to model performance. So if you find ways of improving training efficiency (the…

dannywOct 26, 2026View on HNParent context

I was curious what he was responding to. Per https://authorsguild.org/app/uploads/2026/09/Class-Plaintiff... it was "On July 19, 2019, McCandlish wrote in an OpenAI Slack channel: “We’re not sure if we’re going to release the Foresight LM Scaling paper publicly, but if we do we…

yorwbaSep 27, 2026View on HNParent context

https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... https://papers.nips.cc/paper_files/paper/2012/file/c399862d3... https://arxiv.org/pdf/1512.03385 https://arxiv.org/pdf/1511.07122…

gooobJul 7, 2026View on HNParent context

No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks…

cute_boiJul 7, 2026View on HNParent context

No, well known in the current multiverse branch where we still occasionally use things like math and scientific analysis instead of people’s vibe checks and pelican SVGs. Here’s the paper from OpenAI where Dario himself was a co-author: https://arxiv.org/pdf/2001.08361 > We have…

an0malousJun 20, 2026View on HNParent context

I think some attentive VCs couldve predicted this. Paper from OAI 2020 about scaling laws: https://arxiv.org/pdf/2001.08361 The idea that you can make a system smarter by just putting in more money is very appealing. The earliest OAI coding model is here (Aug 2021)…

sometimelurkerMay 16, 2026View on HNParent context

Better slop. The effect that these systems get better as you scale up [0] is real you know. [0]: https://arxiv.org/pdf/2001.08361

allreduceMar 5, 2026View on HNParent context

Fine, that's fair. I believe the statement that you made is countered by my claim, which is: Theorem. For any tolerance epsilon > 0, there exists a transformer neural network of sufficient size that follows, up to the factor epsilon, the policy that most optimally achieves…

hodgehog11Feb 13, 2026View on HNParent context

It wasn't a lie, it was a misrepresentation of the total cost. It's not hard to calculate the cost of the training though. It takes 6 * active parameters * tokens flops[1]. To get number of seconds you can divide by Flops/s * MFU, where MFU is around 45% for H100 for large…

YetAnotherNickAug 8, 2025View on HNParent context

> The new model seems to prove that longstanding rumors of diminishing returns in training unsupervised-learning LLMs were correct and that the so-called "scaling laws" cited by many for years have possibly met their natural end. The commonly-cited scaling laws[0] predict…

UkvMar 5, 2025View on HNParent context

This is the canonical paper. Nothing I've seen seems to indicate the curves are flattening, you can ask "scaling what" but the trend is clear. https://arxiv.org/pdf/2001.08361

vonneumannstanJan 27, 2025View on HNParent context

You require 6 * parameter * token flops[1] to train LLM. Which means (flop/s of H100 * MFU) / (6 * parameter) token per second. Assuming MFU of 40%, it is (1000 * 10^12 * 0.4) / (6 * 10^9) token/sec = 67,000 token/sec. This repo[2] by Meta achieves 48% MFU, or 80k token/second.…

YetAnotherNickNov 2, 2024View on HNParent context

>if your methodology is flawed you aren't gonna get any more significant returns. The problem with this statement is that predictions made about scaling 5 years ago have held true[1]. We keep adding parameters, adding compute, and the models keep getting more capable. The flaws…

Workaccount2Oct 11, 2024View on HNParent context

>I think the data will show that it is going to take logarithmic costs in either train-time or inference-time for linear or even sub-linear improvements. The data already shows this, it's a well-established result: https://arxiv.org/abs/2001.08361 . Well, less established for…

logicchainsSep 26, 2024View on HNParent context

> New LLMs don't even rely that much on that aforementioned older architecture Don’t they all indicate being based on the transformer architecture? > not entirely because of transformers but because of the hardware Kaplan et al. 2020[0] (figure 7, §3.2.1) shows that LSTMs, the…

espadrineSep 26, 2024View on HNParent context

a backup just in case https://nlp.seas.harvard.edu/annotated-transformer/ https://scottaaronson.blog/?p=762 https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://arxiv.org/pdf/1409.2329.pdf…

is_trueMay 18, 2024View on HNParent context

> Why? I’m utterly baffled by this. Yes, me too! I am surprised people are unaware of the empirical power-scale law. [1] Currently, Anthropic's new model has proven it by being better than GPT-4 on some tasks. Yet, the old people are saying it is okay. They are saying…

findingMeaningMar 10, 2024View on HNParent context

Number go up. More seriously, it looks like a potential reduction in the cost to train a neural network. You can think of every meaningful step forward in deep learning as a reduction in the cost of training, or an improvement in the ability of the signal to propagate.…

hiddencostFeb 1, 2024View on HNParent context

https://openai.com/research/scaling-laws-for-neural-language... https://arxiv.org/abs/2001.08361

dwaltripJan 19, 2024View on HNParent context

The most important paper to understand this issue is "Sacling Laws of Neural Language Models" by Open AI in 2020 [1]. Many consider it the most important paper that predicted the high performance of modern LLMs. This paper shows how the loss decreases when you increase the model…

duchenneDec 15, 2023View on HNParent context

More data, more parameters, more compute all result in a better model per "Scaling Laws for Neural Language Models" https://browse.arxiv.org/pdf/2001.08361v1.pdf Largeness is a valid goal.

lappaOct 19, 2023View on HNParent context

i highlighted this last night which seems to be making the rounds - https://twitter.com/swyx/status/1699369076529971545 Alec's CV (https://www.linkedin.com/in/alecradford/) seems to be: - 2011-2016 BSc Eng from Olin College - 2013+ started a data/AI consultancy as a sophomore…

swyxSep 6, 2023View on HNParent context

If you want theory there’s this: https://arxiv.org/abs/2001.08361 (I haven’t actually read this but I know roughly what it’s about) It’s saying that so far the abilities of an LLMs have scaled up with its parameter count and training data size. Of course there’s no way to be…

circuit10Aug 16, 2023View on HNParent context

There’s a theoretical, but impractical, way: for a given model, each possible set of weight/bias values yields a specific loss value when ran against the full corpus. There’s at least one set of weight values which minimizes it, for which the idealized bit-per-byte entropy can…

espadrineJul 26, 2023View on HNParent context

I love bashing Intel as much as the other person, but looking at the number of tokens LLaMA[1] was trained on (1,400B tokens), the number of parameters of GPT-3[2] vs. the number of tokens it was trained on (175B params vs. 300B), it's not like the announced 1,000B param model…

ppaattrriicckkMay 22, 2023View on HNParent context

The ones I had in mind were the newer 'Chinchilla' scaling laws (https://arxiv.org/pdf/2203.15556.pdf) vs. the older 'Kaplan' scaling laws (https://arxiv.org/pdf/2001.08361.pdf)

ftxbroApr 15, 2023View on HNParent context

This is exactly what happened with large language models: they are quite useless until a certain number of parameters, after which their abilities quickly become impressive (read more here: http://arxiv.org/abs/2001.08361)

blackbear_Mar 20, 2023View on HNParent context

https://arxiv.org/pdf/2001.08361.pdf. See the C_forward formula approxiamtion.

vishal0123Mar 2, 2023View on HNParent context

hey thanks for your reply. umm...so does OpenAI. In fact this is OpenAI discovery from [1]: >Convergence is inefficient: When working within a fixed compute budget C but without any other restric- tions on the model size N or available data D, we attain optimal performance by…

sandGorgonFeb 25, 2023View on HNParent context

>If you took GPT3's architecture and scaled it down to the size and training set of a typical auto-predict, it would produce near identical results. This is almost certainly not true. The number of parameters is an important feature related to the quality of output. If you…

hackinthebochsFeb 14, 2023View on HNParent context

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. and multiple follow-ups

EvgeniyZhFeb 3, 2023View on HNParent context

They talk specifically of scaling laws investigated in papers like OpenAI's "Scaling Laws for Neural Language Models" (2020)[1] and Deepmind's "An empirical analysis of compute-optimal large language model training" (2022)[2] which showed that, as far as we could see at the…

airgapstopgapJan 28, 2023View on HNParent context

Scaling Laws for Neural Language Models - https://arxiv.org/abs/2001.08361

visargaApr 4, 2022View on HNParent context

There has been some effort to quantify "AI scaling laws": ie how much performance increases as we scale up the resources involved. See section 1.2 of https://arxiv.org/pdf/2001.08361.pdf My main takeaway from that paper is that a 2x increase in training cost improves performance…

kmodMar 19, 2022View on HNParent context

> What's worse, the trend to supersize language models is never justified, either theoretically (ha ha) or empirically in the relevant literature What? That's absurd. Large language models are motivated by empirical scaling law. It is actually better justified than other ML…

sanxiynOct 12, 2021View on HNParent context

First, congratulations to you and the other authors - this is a very informative and accessible paper with impressive results. > On both datasets, we find that synthesis performance scales log-linearly with model size This "scaling law" observation is starting to become a major…

criticaltinkerAug 18, 2021View on HNParent context

The author keeps referring to the "PSPACE" complexity of chess and Go in the context of AlphaGo; this is incorrect - these games are only PSPACE for arbitrary large board size N, at fixed board size as actually used by humans and current AI they are just constant O(1)…

InufuFeb 22, 2021View on HNParent context

This is not true. The OpenAI team only trained one full-sized GPT-3, and conducted their hyperparameter sweep on significantly smaller models (see: https://arxiv.org/abs/2001.08361). The compute savings from not having to do the hyperparameter sweep are negligible and do not…

jne2356Jan 18, 2021View on HNParent context

OpenAI only trained the full sized GPT-3 once. Hyperparameter sweep was conducted on significantly smaller models (see: https://arxiv.org/abs/2001.08361)

jne2356Jan 18, 2021View on HNParent context

There's actually a paper that attempts to quantify the scaling characteristics of NLP models from early this year. (https://arxiv.org/abs/2001.08361) Even if scaling alone ends up solving everything (I doubt it), I'd still feel that very significant improvements ought to be…

ReelinSep 4, 2020View on HNParent context

> The way I interpret the lottery ticket hypothesis is that you don't actually need the full sized networks (with their structure and parameters) in order to perform well at some tasks Actually that's not the point. Pruning typically results in networks that still perform well…

ReelinSep 4, 2020View on HNParent context

For example, look at OpenAI's latest paper on scaling Transformers "Scaling Laws for Neural Language Models" https://arxiv.org/abs/2001.08361 , Kaplan et al 2020: larger models are better, in the entire range they test up to billion-parameter models, in pretty much every way…

gwernJan 25, 2020View on HNParent context