ZeRO-Offload: Democratizing Billion-Scale Model Training · HackerPapers discussions

ZeRO-Offload: Democratizing Billion-Scale Model Training — mentioned by sighingnow

Tue, 10 Oct 2023 12:43:57 GMT

The pipedream-2bw paper[1] and the zero-offload paper[2] both show that 1-step delayed asynchronous gradient update doesn't affect the convergence (and perplexity) while improve the training efficiency (by fully utilize the bubbles in pipeline parallelism) at a large margin. However, both the Megatron-LM[3] and the DeepSpeed[4] don't use pipedream-2bw scheduling. Could anyone share me some insights or ideas about why such an efficient scheduling scheme doesn't get popular in the LLM pretraining community? Does it suffer convergence/accuracy issue in practice? Or are there any other concerns…

Paper and discussion history. A mention is not an endorsement.

ZeRO-Offload: Democratizing Billion-Scale Model Training — mentioned by treprinum

Wed, 31 May 2023 17:48:39 GMT

That's been done already. See DeepSpeed ZeRO NVMe offload: https://arxiv.org/abs/2101.06840

Paper and discussion history. A mention is not an endorsement.

ZeRO-Offload: Democratizing Billion-Scale Model Training — mentioned by BadInformatics

Sun, 14 Mar 2021 01:02:38 GMT

Having followed this DeepSpeed stuff for a little while, the ZeRO paper is probably as close as you can get to an ELI5 because there's no singular brilliant idea behind this. Most of the ideas have been explored already (see e.g. the PyTorch DDP paper), but ZeRO takes them to their logical conclusion by throwing a TON of engineering work into the equation. For example, they implement custom fused kernels on CPU/GPU and a hand-vectorized Adam implementation. I found that this earlier blog post [2] has a much better deep dive (with decent animations and more) into the underlying architecture.…

Paper and discussion history. A mention is not an endorsement.

ZeRO-Offload: Democratizing Billion-Scale Model Training — mentioned by p1esk

Sat, 13 Mar 2021 22:56:43 GMT

They did all that before: https://arxiv.org/abs/2101.06840, but they could only fit a model with 13B weights on a single V100.

Paper and discussion history. A mention is not an endorsement.

ZeRO-Offload: Democratizing Billion-Scale Model Training — HN submission

Wed, 03 Feb 2021 17:06:59 GMT

Zero-Offload: Democratizing Billion-Scale Model Training

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.