MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies · HackerPapers discussions

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies — mentioned by kouteiheika

Fri, 23 May 2025 20:03:30 GMT

> Let's look at some of the most important ones that have been developed over the years and try to implement the basic ideas as succinctly as possible. One big architectural tweak that comes to mind and isn't in the article is QK norm: https://arxiv.org/pdf/2010.04245 > Cosine Schedule A lot (most?) of new training runs actually don't use cosine schedule anymore; instead they keep the learning rate constant and only decay it at the very end, which gives equivalent or better results. See: https://arxiv.org/pdf/2405.18392 https://arxiv.org/pdf/2404.06395 > There is a highly optimized…

Paper and discussion history. A mention is not an endorsement.

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies — HN submission

Sat, 13 Apr 2024 23:32:01 GMT

Chinchilla Debunked

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies — HN submission

Wed, 10 Apr 2024 12:25:05 GMT

MiniCPM: Potential of Small Language Models W Scalable Training Strategies

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.