Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — mentioned by sailingparrot

Sat, 31 Jan 2026 01:36:34 GMT

> Nvidia has been using its newfound liquid funds to train its own family of models Nvidia has always had its own family of models, it's nothing new and not something you should read too much into IMHO. They use those as template other people can leverage and they are of course optimized for Nvidia hardware. Nvidia has been training models in the Megatron family as well as many others since at least 2019 which was used as blueprint by many players. [1] [1] https://arxiv.org/abs/1909.08053

Paper and discussion history. A mention is not an endorsement.

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — mentioned by lucidrains

Fri, 25 Oct 2019 01:51:57 GMT

Yes. The last one was 8.3B https://arxiv.org/pdf/1909.08053.pdf

Paper and discussion history. A mention is not an endorsement.

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism · HackerPapers discussions

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — mentioned by sailingparrot

Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — mentioned by lucidrains