HackerPapers

arXiv:2312.15166cs.CL

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling

Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, Sunghun Kim

Abstract

We introduce SOLAR 10.7B, a large language model (LLM) with 10.7 billion parameters, demonstrating superior performance in various natural language processing (NLP) tasks. Inspired by recent efforts to efficiently up-scale LLMs, we present a method for scaling LLMs called depth up-scaling (DUS), which encompasses depthwise scaling and continued pretraining. In contrast to other LLM up-scaling methods that use mixture-of-experts, DUS does not require complex changes to train and inference efficiently. We show experimentally that DUS is simple yet effective in scaling up high-performance LLMs from small ones. Building on the DUS model, we additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for instruction-following capabilities, surpassing Mixtral-8x7B-Instruct. SOLAR 10.7B is publicly available under the Apache 2.0 license, promoting broad access and application in the LLM field.

arXiv comments
accepted to NAACL 2024 Industry Track
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 4 threads, from Dec 2023 to Mar 2026. Found in 1 HN submission from Dec 2023 to Dec 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Show HN: Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No training 1 comment

    • Reminds me of Solar 10.7B, which was a very good model for its size ~2 year ago and the "Depth Up-Scaling" technique behind it. Although, that involved continued training after repeating the layers. https://arxiv.org/abs/2312.15166

      woadwarrior01View on HNParent context

  2. Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUs 1 comment

    • Amazing write up and i wish more people showed the process for discovery which is often even more interesting than the result itself Still the result is really interesting being able to stack abstract reasoning and get better performance and the heat maps to show the prob…

      imranqView on HNParent context

  3. Fine-tuning LLMs is a waste of time 1 comment

    • In general this is of course an active area of research, but yes, you can do something that and people have done it successfully[1] by adding extra layers to an existing model and then continuing to train it. You have to be careful about the "same data" part though; ideally you…

      kouteiheikaView on HNParent context

  4. Solar 10.7B: Elevating AI, Effortlessly 1 comment

    • DUS(Depth Up-Scaling)-driven SOLAR 10.7B outshines peers, setting a new standard in AI performance. https://arxiv.org/abs/2312.15166

      haebomView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.