Distributed Deep Learning in Open Collaborations · HackerPapers discussions

Distributed Deep Learning in Open Collaborations — mentioned by anuramat

Mon, 20 Jan 2025 23:08:12 GMT

I think you're looking for federated/distributed learning. From what I understand, nobody quite figured it out yet. Relevant 2021 NIPS paper: <https://arxiv.org/abs/2106.10207> On difference between federated and distributed: <https://en.wikipedia.org/w/index.php?title=Federated_learnin...>

Paper and discussion history. A mention is not an endorsement.

Distributed Deep Learning in Open Collaborations — mentioned by 127361

Tue, 16 Jan 2024 12:29:03 GMT

"This paper presents a distributed model-parallel training framework that enables training large neural networks on small CPU clusters with low Internet bandwidth." Low bandwidth being <1Gbps. They've also tested it with GPUs as well. https://arxiv.org/pdf/2201.12667 Maybe there's the possibility of a completely new AI architecture that can still be efficiently trained when there are very low bandwidth connections between nodes? Specifically targeting this use case would make sense, given all the millions of underutilized GPUs out there in peoples' desktop computers. Also…

Paper and discussion history. A mention is not an endorsement.

Distributed Deep Learning in Open Collaborations — mentioned by sbierwagen

Wed, 27 Dec 2023 18:24:35 GMT

By "SETI" I assume you mean the SETI@Home distributed computing project. There's a two-way market where you can rent out your GPU here: https://vast.ai/ If you want to donate your GPU you could set your price to $0.00 I guess. Yandex and Hugging Face trained a couple nets using distributed GPUs. Their lessons learned paper is here: https://arxiv.org/abs/2106.10207 Takeaway: using consumer GPUs on the other end of consumer internet connections means you end up very I/O limited. 50 megabit upload on a cable modem doesn't compare well with 50000 megabit NVswitch links in a proper GPU cluster. As…

Paper and discussion history. A mention is not an endorsement.

Distributed Deep Learning in Open Collaborations — HN submission

Mon, 21 Jun 2021 20:41:27 GMT

DeDLOC – Distributed Deep Learning in Open Collaborations

Yandex, Hugging Face, HSE, Moscow Institute of Physics and Technology, University of Toronto and the Vector Institute propose a method to share compute power and ran a decentralized collaborative training of an Albert-like model in Bengali. https://arxiv.org/abs/2106.10207 https://github.com/yandex-research/DeDLOC https://twitter.com/m_ryabinin/status/1406988626374762503

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Distributed Deep Learning in Open Collaborations — HN submission

Mon, 21 Jun 2021 01:57:21 GMT

Distributed Deep Learning in Open Collaborations

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.