Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts — mentioned by spyder

Fri, 31 Mar 2023 21:38:53 GMT

Learning@Home using Decentralized Mixture-of-Expert models: https://learning-at-home.github.io/ https://training-transformers-together.github.io/ https://arxiv.org/abs/2002.04013

Paper and discussion history. A mention is not an endorsement.

Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts — mentioned by mryab

Sat, 30 May 2020 19:41:52 GMT

Here is a recent paper (disclaimer: I am the first author) named "Learning@home" which proposes something along these lines. Basically, we develop a system that allows you to train a network with thousands of "experts" distributed across hundreds or more of consumer-grade PCs. You don't have to fit 700GB of parameters on a single machine and there is significantly less network delay as for synchronous model parallel training. The only thing you sacrifice is the guarantee that all the batches will be processed by all required experts. You can read it on ArXiv https://arxiv.org/abs/2002.04013v1…

Paper and discussion history. A mention is not an endorsement.

Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts · HackerPapers discussions

Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts — mentioned by spyder

Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts — mentioned by mryab