SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling · HackerPapers discussions

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling — mentioned by woadwarrior01

Thu, 19 Mar 2026 00:55:05 GMT

Reminds me of Solar 10.7B, which was a very good model for its size ~2 year ago and the "Depth Up-Scaling" technique behind it. Although, that involved continued training after repeating the layers. https://arxiv.org/abs/2312.15166

Paper and discussion history. A mention is not an endorsement.

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling — mentioned by imranq

Tue, 10 Mar 2026 20:05:48 GMT

Amazing write up and i wish more people showed the process for discovery which is often even more interesting than the result itself Still the result is really interesting being able to stack abstract reasoning and get better performance and the heat maps to show the prob results The academic literature seems to be catching up: - *[SOLAR / DUS (Kim et al., 2023)](https://arxiv.org/abs/2312.15166)* — duplicated transformer layers to build a 10.7B model that outperformed 30B parameter baselines. - *[The Curse of Depth (2025)](https://arxiv.org/abs/2502.05795)* — explains why this works: Pre-LN…

Paper and discussion history. A mention is not an endorsement.

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling — mentioned by kouteiheika

Thu, 12 Jun 2025 03:32:50 GMT

In general this is of course an active area of research, but yes, you can do something that and people have done it successfully[1] by adding extra layers to an existing model and then continuing to train it. You have to be careful about the "same data" part though; ideally you want to train once on unique data[2] as excessive duplication can harm the performance of the model[3], although if you have limited data a couple of training epochs might be safe and actually improve the performance of the model[4]. [1] -- https://arxiv.org/abs/2312.15166 [2] -- https://arxiv.org/abs/1906.06669 [3]…

Paper and discussion history. A mention is not an endorsement.

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling — HN submission

Wed, 27 Dec 2023 14:03:36 GMT

Solar 10.7B

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling — mentioned by haebom

Wed, 27 Dec 2023 07:03:19 GMT

DUS(Depth Up-Scaling)-driven SOLAR 10.7B outshines peers, setting a new standard in AI performance. https://arxiv.org/abs/2312.15166

Paper and discussion history. A mention is not an endorsement.