Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM · HackerPapers discussions

Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM — mentioned by magicalhippo

Mon, 27 Jan 2025 22:13:45 GMT

Not my field, but from this[1] blog post which references this[2] paper, it would seem so. Note the optimal approach are a bit different between training and inference. Also note that several of the approaches rely on batching multiple requests (prompts) in order to exploit the parallelism, so won't see the same gains if fed only a single prompt at a time. [1]: https://medium.com/@plienhar/llm-inference-series-4-kv-cachi... [2]: https://arxiv.org/abs/2104.04473

Paper and discussion history. A mention is not an endorsement.

Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM — mentioned by depr

Thu, 04 May 2023 08:38:51 GMT

Not as much as it took GPT to process all its input. >Let us consider the GPT-3 model with 𝑃 =175 billion parameters as an example. This model was trained on 𝑇 = 300 billion tokens. On 𝑛 = 1024 A100 GPUs using batch-size 1536, we achieve 𝑋 = 140 teraFLOP/s per GPU. As a result, the time required to train this model is 34 days. https://arxiv.org/pdf/2104.04473.pdf I'm not sure expressing brain capacity in FLOPs makes much sense, but I'm sure if it can be expressed in FLOPs, the amount of FLOPs going to learning for a normal human is less than that.

Paper and discussion history. A mention is not an endorsement.