Stable LM 2 1.6B Technical Report · HackerPapers discussions

Stable LM 2 1.6B Technical Report — mentioned by BryanLegend

Fri, 08 Mar 2024 17:41:56 GMT

We’ve published our technical report on Stable LM 2 1.6B; our state-of-the-art small language model trained on multilingual data. This in-depth study covers the training methods utilized for both the base and instruction-tuned versions of the model. Leading in the OpenLLM leaderboard across multiple languages including English, German, Spanish, French, Italian, Dutch, and Portuguese, our model achieves top scores when ranked against larger sized models such as Google’s Gemma 2. We've also introduced a multi-stage infinite Learning Rate Scheduler that enables us to adjust the model's learning…

Paper and discussion history. A mention is not an endorsement.

Stable LM 2 1.6B Technical Report — mentioned by omnipotent_i

Thu, 29 Feb 2024 06:43:10 GMT

StableLM 1.6B is a strong multilingual LM trained for 2T tokens of multilingual dataset. It offers strong performance compared to other models of the same size range including Google’s Gemma, Microsoft’s Phi2 and ByteDance’s Qwen. The technical report discloses all of the necessary details to reproduce the training procedure including the data mix. Training details: RoPE, LayerNorm, No MLP bias terms, but retains for QKV terms. Global Batch Size - 8M tokens MFU during training - 54.5% Tokenizer - based out of OpenAI's tiktoken's CL100k_base with vocab size ~ 100k. Paper shows a new hybrid LR…

Paper and discussion history. A mention is not an endorsement.

Stable LM 2 1.6B Technical Report — HN submission

Thu, 29 Feb 2024 06:43:10 GMT

StableLM 1.6B Technical Report – includes all data, training, strategy

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.