Asynchronous Local-SGD Training for Language Modeling · HackerPapers discussions

Asynchronous Local-SGD Training for Language Modeling — mentioned by blt

Wed, 07 May 2025 21:30:09 GMT

I agree with this part of your response: If we were to require that distributed training generates the exact same sequence of weight updates as serial SGD on a single machine, then we would need a barrier like that. However, there is a lot of research on distributed optimization that addresses this issue by relaxing the "exactly equivalent to serial SGD" requirement, including classic papers [1] and more recent ones [2]. Basically, there are two properties of ML optimization that save us: 1) the objective is a huge summation over many small objectives (the losses for each training data…

Paper and discussion history. A mention is not an endorsement.

Asynchronous Local-SGD Training for Language Modeling — mentioned by gaogao

Tue, 14 Jan 2025 13:20:24 GMT

Yeah, high bandwidth requirements still remaining. Over the past year, more research has looked from fully async to restrained cases that allow for geographically distributed compute. Async Local-SGD goes for a more standard training objective comparable with a lockstep training, https://arxiv.org/abs/2401.09135. imo technique is looking better.

Paper and discussion history. A mention is not an endorsement.

Asynchronous Local-SGD Training for Language Modeling — HN submission

Sat, 20 Jan 2024 11:08:23 GMT

Asynchronous Local-SGD Training for Language Modeling

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.