ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — mentioned by jiofih

Sat, 13 Mar 2021 19:06:18 GMT

You can read the paper here: https://arxiv.org/abs/1910.02054

Paper and discussion history. A mention is not an endorsement.

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — mentioned by choppaface

Mon, 10 Feb 2020 19:52:32 GMT

Even from the paper, it's hard to tell what this library actually does: Section 5 in https://arxiv.org/pdf/1910.02054.pdf The paper talks about parameter partitioning and overlapped communication, but doesn't actually give many details on how those things happen. The library appears to be an implementation of some common algos for solving the 'pebble game,' as explained decently here: https://medium.com/tensorflow/fitting-larger-networks-into-m... The essential point is that: (1) model parallelism is hard to do and has historically been done manually to scale wide models across GPUs (2)…

Paper and discussion history. A mention is not an endorsement.

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models · HackerPapers discussions

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — mentioned by jiofih

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — mentioned by choppaface