HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent · HackerPapers discussions

HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent — mentioned by inciampati

Mon, 23 Aug 2021 09:49:00 GMT

Actually, nope! SGD is embarrassingly parallel when it's HOGWILD! https://arxiv.org/abs/1106.5730. Formally, the hogwild and serial implementations are indistinguishable. This has a lot of citations but it's such an easy trick that no door path beating seems to have happened. The "trick" is just to make model updates atomic, and then each SGD step can happen in parallel without coordination or locks.

Paper and discussion history. A mention is not an endorsement.

HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent — mentioned by yzmtf2008

Tue, 13 Mar 2018 00:11:58 GMT

An example where it's not insanity (while it might seem to be): https://arxiv.org/abs/1106.5730

Paper and discussion history. A mention is not an endorsement.

HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent — mentioned by ebalit

Tue, 30 Aug 2016 15:07:53 GMT

HOGWILD [1] lock-free weight updates are a great example of how forgiving SGD can be. 1 : https://arxiv.org/abs/1106.5730

Paper and discussion history. A mention is not an endorsement.