https://hackerpaper.kaipfister.de/p/2006.09503/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMThttps://news.ycombinator.com/item?id=37831331urn:hackerpapers:mention:2006.09503:37831331Tue, 10 Oct 2023 12:43:57 GMT<p>The pipedream-2bw paper[1] and the zero-offload paper[2] both show that 1-step delayed asynchronous gradient update doesn&#x27;t affect the convergence (and perplexity) while improve the training efficiency (by fully utilize the bubbles in pipeline parallelism) at a large margin. However, both the Megatron-LM[3] and the DeepSpeed[4] don&#x27;t use pipedream-2bw scheduling. Could anyone share me some insights or ideas about why such an efficient scheduling scheme doesn&#x27;t get popular in the LLM pretraining community? Does it suffer convergence/accuracy issue in practice? Or are there any other concerns…</p><p><a href="https://hackerpaper.kaipfister.de/p/2006.09503/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.LG