DiLoCo: Distributed Low-Communication Training of Language Models · HackerPapers discussions

DiLoCo: Distributed Low-Communication Training of Language Models — mentioned by bwfan123

Mon, 12 May 2025 15:32:06 GMT

The most interesting thing I see is the productization of the diloco work done here [1]. If someone can make this scale, then we can say goodbye to expensive backend networking and mainframe-like AI training machinery. [1] https://arxiv.org/abs/2311.08105

Paper and discussion history. A mention is not an endorsement.

DiLoCo: Distributed Low-Communication Training of Language Models — mentioned by bloatedGoat

Wed, 18 Sep 2024 20:07:43 GMT

There are methods that make it feasible to train models over the internet. DiLoCo is one [1] and NousResearch has found a way to improve on that using a method they call DisTro [2]. 1. https://arxiv.org/abs/2311.08105 2. https://github.com/NousResearch/DisTrO?tab=readme-ov-file

Paper and discussion history. A mention is not an endorsement.

DiLoCo: Distributed Low-Communication Training of Language Models — HN submission

Fri, 26 Jul 2024 17:41:05 GMT

DiLoCo: Distributed Low-Communication Training of Language Models

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

DiLoCo: Distributed Low-Communication Training of Language Models — mentioned by hackpert

Wed, 06 Mar 2024 01:38:32 GMT

There has been some interesting work when it comes to distributed training. For example DiLoCo (https://arxiv.org/abs/2311.08105). I also know that Bittensor and nousresearch collaborated on some kind of competitive distributed model frankensteining-training thingy that seems to be going well. https://bittensor.org/bittensor-and-nous-research/ Of course it gets harder as models get larger but distributed training doesn't seem totally infeasible. For example if we were to talk about MoE transformer models, perhaps separate slices of the model can be trained in an asynchronous manner and then…

Paper and discussion history. A mention is not an endorsement.

DiLoCo: Distributed Low-Communication Training of Language Models — mentioned by timschmidt

Thu, 07 Dec 2023 22:06:50 GMT

This seems relevant: https://arxiv.org/abs/2311.08105

Paper and discussion history. A mention is not an endorsement.

DiLoCo: Distributed Low-Communication Training of Language Models — HN submission

Wed, 06 Dec 2023 20:55:05 GMT

DiLoCo: Distributed Low-Communication Training of Language Models

46 points; 14 discussion replies on HN. Paper and source history. A submission is not an endorsement.

DiLoCo: Distributed Low-Communication Training of Language Models — mentioned by pr337h4m

Wed, 22 Nov 2023 21:11:58 GMT

This from DeepMind: DiLoCo: Distributed Low-Communication Training of Language Models - https://arxiv.org/pdf/2311.08105.pdf From the first author on Twitter: "It could quite a big deal for people who don't have access to a colocated cluster of GPUs: e.g. with DiLoCo you could train your model, with data-parallelism, across all GPU providers, looking in real-time for the cheapest price, even if pre-emptable, even across continents" https://twitter.com/Ar_Douillard/status/1724839420820361687

Paper and discussion history. A mention is not an endorsement.

DiLoCo: Distributed Low-Communication Training of Language Models — HN submission

Wed, 22 Nov 2023 16:20:37 GMT

Distributed Low-Communication Training of Language Models

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

DiLoCo: Distributed Low-Communication Training of Language Models — HN submission

Tue, 21 Nov 2023 22:31:53 GMT

DiLoCo: Distributed Low-Communication Training of Language Models

5 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.