On the Generalization Benefit of Noise in Stochastic Gradient Descent — mentioned by blackbear_

Sun, 02 Nov 2025 07:51:40 GMT

Two thoughts: > how important is computing the exact gradient using calculus Normally the gradient is computed with a small "minibatch" of examples, meaning that on average over many steps the true gradient is followed, but each step individually never moves exacty along the true gradient. This noisy walk is actually quite beneficial for the final performance of the network https://arxiv.org/abs/2006.15081 , https://arxiv.org/abs/1609.04836 so much so that people started wondering what is the best way to "corrupt" this approximate gradient even more to improve performance…

Paper and discussion history. A mention is not an endorsement.

On the Generalization Benefit of Noise in Stochastic Gradient Descent · HackerPapers discussions

On the Generalization Benefit of Noise in Stochastic Gradient Descent — mentioned by blackbear_