Decentralized Training of Foundation Models in Heterogeneous Environments · HackerPapers discussions

Decentralized Training of Foundation Models in Heterogeneous Environments — mentioned by arilotter

Wed, 28 Aug 2024 18:19:36 GMT

That's not something that DisTrO solves, but there's plenty of research in that area! See https://arxiv.org/abs/2301.11913 , https://arxiv.org/abs/2206.01288 , https://arxiv.org/abs/2304.11277 etc :)

Paper and discussion history. A mention is not an endorsement.

Decentralized Training of Foundation Models in Heterogeneous Environments — mentioned by mryab

Mon, 20 Feb 2023 18:55:24 GMT

There is! See https://petals.ml/ for inference of models like BLOOM-176B over the internet or https://arxiv.org/abs/2301.11913 and https://arxiv.org/abs/2206.01288 that show you how to do pretraining from scratch in the same setting. Disclaimer: I'm a coauthor of these systems (including the one in OP)

Paper and discussion history. A mention is not an endorsement.

Decentralized Training of Foundation Models in Heterogeneous Environments — mentioned by Vetch

Wed, 30 Nov 2022 23:16:47 GMT

For inference, the best models are so large they won't fit in System RAM. GPT@home is not going to make a difference in that scenario. For training such large models, data parallelism is no longer sufficient and tensor/pipeline parallelism is required. The problem is communication bottlenecks, differing device/network speeds and massive data transfer requirements become serious enough issues to kill any naive distributed training across the internet approach. Deep learning companies use fancy 100Gbps+ connections, do kernel hacking and use homogeneous hardware and it's still a serious…

Paper and discussion history. A mention is not an endorsement.

Decentralized Training of Foundation Models in Heterogeneous Environments — HN submission

Thu, 09 Jun 2022 17:10:55 GMT

Decentralized Training of Foundation Models in Heterogeneous Environments

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.