https://hackerpaper.kaipfister.de/p/2006.15704/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMThttps://news.ycombinator.com/item?id=26451750urn:hackerpapers:mention:2006.15704:26451750Sun, 14 Mar 2021 01:02:38 GMT<p>Having followed this DeepSpeed stuff for a little while, the ZeRO paper is probably as close as you can get to an ELI5 because there&#x27;s no singular brilliant idea behind this. Most of the ideas have been explored already (see e.g. the PyTorch DDP paper), but ZeRO takes them to their logical conclusion by throwing a TON of engineering work into the equation. For example, they implement custom fused kernels on CPU/GPU and a hand-vectorized Adam implementation. I found that this earlier blog post [2] has a much better deep dive (with decent animations and more) into the underlying architecture.…</p><p><a href="https://hackerpaper.kaipfister.de/p/2006.15704/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.DC