signSGD: Compressed Optimisation for Non-Convex Problems — mentioned by f_devd

Sat, 12 Oct 2024 17:22:33 GMT

It has been more formally studied in signSGD[0], and empirically it's comparable to Adam in terms of behavior. [0]: https://arxiv.org/pdf/1802.04434

Paper and discussion history. A mention is not an endorsement.

signSGD: Compressed Optimisation for Non-Convex Problems — mentioned by mtcrawshaw

Sat, 18 Feb 2023 16:16:36 GMT

Your formula is almost right but there is a slight difference. Namely the third line of Program 1 in their paper is absent from your statement, which will change the value of the momentum variable for subsequent steps. To be fair, I don't think that this operation will fundamentally change the behavior of the algorithm. In fact, I don't think their algorithm has a fundamentally different behavior than SignSGD with momentum, whose convergence behavior is well-understood [1, 2, 3]. Actually, the algorithm you described is equivalent to SignSGD with momentum. In the algorithm you described, m is…

Paper and discussion history. A mention is not an endorsement.

signSGD: Compressed Optimisation for Non-Convex Problems · HackerPapers discussions

signSGD: Compressed Optimisation for Non-Convex Problems — mentioned by f_devd

signSGD: Compressed Optimisation for Non-Convex Problems — mentioned by mtcrawshaw