Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations — mentioned by kouteiheika

Fri, 23 May 2025 20:03:30 GMT

> Let's look at some of the most important ones that have been developed over the years and try to implement the basic ideas as succinctly as possible. One big architectural tweak that comes to mind and isn't in the article is QK norm: https://arxiv.org/pdf/2010.04245 > Cosine Schedule A lot (most?) of new training runs actually don't use cosine schedule anymore; instead they keep the learning rate constant and only decay it at the very end, which gives equivalent or better results. See: https://arxiv.org/pdf/2405.18392 https://arxiv.org/pdf/2404.06395 > There is a highly optimized…

Paper and discussion history. A mention is not an endorsement.

Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations · HackerPapers discussions

Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations — mentioned by kouteiheika