DeMo: Decoupled Momentum Optimization · HackerPapers discussions

DeMo: Decoupled Momentum Optimization — mentioned by aimanbenbaha

Tue, 14 Jan 2025 13:20:12 GMT

This bottleneck right here is why Open Source is presented with a golden plate opportunity to lead the training of cutting edge models. Federated learning breaks the barrier to entry and expands the ecosystem allowing more participants to share compute and/or datasets for small players to train models. DiLoCo introduced by Douillard minimizes communication overhead by averaging weight updates. What this article misses though is that despite this, each GPU in the distributed cluster still needs to have enough VRAM to load the entire copy of the model to complete the training process. That's…

Paper and discussion history. A mention is not an endorsement.

DeMo: Decoupled Momentum Optimization — HN submission

Tue, 03 Dec 2024 03:07:36 GMT

Demo: Decoupled Momentum Optimization

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

DeMo: Decoupled Momentum Optimization — HN submission

Mon, 02 Dec 2024 18:37:09 GMT

Demo: Decoupled Momentum Optimization – training neural networks in parallel

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.