SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient · HackerPapers discussions

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient — mentioned by arilotter

Wed, 28 Aug 2024 18:19:36 GMT

That's not something that DisTrO solves, but there's plenty of research in that area! See https://arxiv.org/abs/2301.11913 , https://arxiv.org/abs/2206.01288 , https://arxiv.org/abs/2304.11277 etc :)

Paper and discussion history. A mention is not an endorsement.

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient — HN submission

Mon, 03 Jun 2024 00:59:18 GMT

Swarm Parallelism: Training Large Models on Poorly Connected Devices

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient — mentioned by 1827163

Mon, 22 May 2023 20:25:35 GMT

It might not require specialized hardware in the future as GPUs get more powerful and we have other techniques such as LoRA for fine-tuning the models. We might see a distributed training [1] effort harnessing thousands of gamer GPUs worldwide, as well. All of this powered by open source software. Also there could be advances in the training software making it vastly more efficient. 1. https://arxiv.org/pdf/2301.11913.pdf

Paper and discussion history. A mention is not an endorsement.

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient — mentioned by mryab

Mon, 20 Feb 2023 18:55:24 GMT

There is! See https://petals.ml/ for inference of models like BLOOM-176B over the internet or https://arxiv.org/abs/2301.11913 and https://arxiv.org/abs/2206.01288 that show you how to do pretraining from scratch in the same setting. Disclaimer: I'm a coauthor of these systems (including the one in OP)

Paper and discussion history. A mention is not an endorsement.

SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient — mentioned by zone411

Sat, 04 Feb 2023 19:49:14 GMT

Long story short, training requires intensive device-to-device communication. Distributed training is possible in theory but so inefficient that it's not worth it. Here is a new paper that looks to be the most promising approach yet: https://arxiv.org/abs/2301.11913

Paper and discussion history. A mention is not an endorsement.