On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima · HackerPapers discussions

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — mentioned by blackbear_

Sun, 02 Nov 2025 07:51:40 GMT

Two thoughts: > how important is computing the exact gradient using calculus Normally the gradient is computed with a small "minibatch" of examples, meaning that on average over many steps the true gradient is followed, but each step individually never moves exacty along the true gradient. This noisy walk is actually quite beneficial for the final performance of the network https://arxiv.org/abs/2006.15081 , https://arxiv.org/abs/1609.04836 so much so that people started wondering what is the best way to "corrupt" this approximate gradient even more to improve performance…

Paper and discussion history. A mention is not an endorsement.

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — mentioned by chillee

Mon, 18 Dec 2017 22:20:36 GMT

Sorry, my last sentence > Otherwise, models can achieve very good generalization. was probably unclear. When we talk about generalization, we typically have to assume a lack of distributional shift. Otherwise, if the "truly unseen data" really is drawn from a different distribution (say one where all cats are now called "dogs" and all dogs are now called "cats"), we can get arbitrarily poor performance. As for the flat minima part, lemme pull up some papers from my notes. https://arxiv.org/abs/1609.04836 showed larger batch sizes lead to sharper minima. > ... and as is well known, sharp…

Paper and discussion history. A mention is not an endorsement.

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — mentioned by chillee

Wed, 13 Dec 2017 05:10:04 GMT

Really don't think that's the best paper to say "sheds quite a bit of light on this". That paper has been somewhat controversial since it came out. I think https://arxiv.org/abs/1609.04836 is seminal in showing unsharp minima = generalization, the parent's paper is good for showing that gradient descent over non-convex surfaces works fine, https://arxiv.org/abs/1611.03530 is landmark for kicking off this whole generalization business (mainly shows that traditional models of generalization, namely VC dimension and ideas of "capacity" don't make sense for neural nets).

Paper and discussion history. A mention is not an endorsement.

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — mentioned by IanCal

Tue, 13 Jun 2017 09:35:17 GMT

> shouldn't a larger batch_size inherently provide more information about the optimal direction of the update? Not necessarily, since a batch gradient output (as I understand it, and at least used to code it) all gets averaged together. Consider standing in a valley with two equal hills either side of you. If you were to try one direction and see that climbing that way helps, you'd take a step that way. Then the next step would keep taking you up that hill. Now, if you batched together two direction tests, what would happen? You'd average together your left and right and end up with moving…

Paper and discussion history. A mention is not an endorsement.

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — mentioned by mijoharas

Thu, 23 Mar 2017 12:22:00 GMT

My understanding from this article is that for a greater GPU ram size you will need to increase the minibatch size which has a negative impact on the models ability to generalize[0]. [0] https://arxiv.org/abs/1609.04836

Paper and discussion history. A mention is not an endorsement.

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima — mentioned by ogrisel

Tue, 22 Nov 2016 21:58:24 GMT

Then the total batch size is growing with the number of GPUs and the convergence might be impacted both in terms of speed and solution quality (e.g. https://arxiv.org/abs/1609.04836 ). I could believe you if tell you me that the validation loss and test accuracy of the large distributed model remains as good as the sequential, single GPU model after the same total number of epochs but this is not a given and if it's not the case I would find those benchmarks deceptive.

Paper and discussion history. A mention is not an endorsement.