Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift · HackerPapers discussions

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — HN submission

Fri, 27 Feb 2026 01:01:10 GMT

Batch Normalization: Accelerating Deep Network Training (2015)

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — mentioned by radq

Mon, 19 Jun 2023 07:21:26 GMT

We were missing two architecture patterns that were needed to get deeper nets to converge: residual nets [1] which solved gradient propagation, and batch normalization [2] which solved initialization. [1] Residual nets (2015): https://arxiv.org/abs/1512.03385 [2] Batch normalization (2015): https://arxiv.org/abs/1502.03167

Paper and discussion history. A mention is not an endorsement.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — mentioned by walrus

Thu, 02 Mar 2017 18:15:42 GMT

I think they're referring to the momentum parameter at [1]. The exponential moving average (EMA) of the batch mean/variance is used in the batch normalizing transform (Algorithm 1 in [2]). The momentum ranges from 0 to 1. If it's close to 1, which the default of 0.99 is, the EMA of the batch mean/variance will change slowly across batches. If it's close to 0, the EMA will be close to the mean/variance of the current batch. The EMA acts as a low-pass filter. With a momentum close to 1, the EMA changes slowly, filtering out high frequencies and leaving only frequencies close to DC. Note that…

Paper and discussion history. A mention is not an endorsement.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — mentioned by sja

Wed, 24 Aug 2016 19:17:08 GMT

Good list! I think it's important to note that this article is (intentionally) focused on modern CNN architectures, and not "deep learning" in general. I'd also add in the following "technique" articles: Geoff Hinton et al.'s dropout paper[0] and Loffe and Szegedy's Batch Normalization paper[1]. I don't think there's been enough time for the dust to settle, but I'm excited about the possibilities Stochastic Depth[2] could offer, too. [0]: http://arxiv.org/abs/1207.0580 [1]: http://arxiv.org/abs/1502.03167 [2]: http://arxiv.org/abs/1603.09382

Paper and discussion history. A mention is not an endorsement.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — mentioned by walrus

Wed, 13 Apr 2016 05:38:14 GMT

I'm not qualified to answer this, but I will anyway. To "operate" neural networks (as opposed to writing a framework for them), you need to know the building blocks. There are basic blocks like fully connected layers, convolutions, and nonlinear activations. Beyond those, there are higher level building blocks like LSTMs[1], gated recurrent units[2], highway layers[3], batch normalization[4], and residual blocks[5] that are made up of simpler blocks. Learning what these do and when it's appropriate to use them requires following current literature. Operating neural networks requires some…

Paper and discussion history. A mention is not an endorsement.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — mentioned by twelfthnight

Thu, 07 May 2015 12:15:45 GMT

http://image-net.org/challenges/LSVRC/2012/results.html One team at 15% http://www.image-net.org/challenges/LSVRC/2014/results Most teams are below 15%, GooLeNet is at 6% http://www.image-net.org/challenges/LSVRC/2014/results Microsoft is now below 5% http://arxiv.org/pdf/1502.03167.pdf Google is now below 5% I can't really argue whether that's exponential or not, but it amazing progress in a short amount of time.

Paper and discussion history. A mention is not an endorsement.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — HN submission

Fri, 13 Feb 2015 23:05:40 GMT

Batch Normalization: Accelerating Deep Network Training [pdf]

134 points; 38 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift — HN submission

Thu, 12 Feb 2015 05:23:25 GMT

Ioffe and Szegedy get 4.8% top-5 test on ImageNet(beating recent MSR result)

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.