Revisiting Small Batch Training for Deep Neural Networks · HackerPapers discussions

Revisiting Small Batch Training for Deep Neural Networks — mentioned by jcreinhold

Mon, 06 Jul 2020 17:17:40 GMT

Regarding smaller batch sizes and batch normalization: Have you found another normalization layer to work better for small batch sizes? I agree that the mean and variance of the small batches won't be representative of the true mean and variance, but in practice, I've used batch norm for small batch sizes successfully (e.g., <8). In medical imaging, due to memory constraints, I commonly see batch sizes of 2 (or even 1, although it's not really "batch norm" at that point). The paper "Revisiting small batch training for deep neural networks" [1] discusses the benefits of small batch sizes even…

Paper and discussion history. A mention is not an endorsement.

Revisiting Small Batch Training for Deep Neural Networks — HN submission

Mon, 01 Jun 2020 21:10:47 GMT

Revisiting Small Batch Training for Deep Neural Networks

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Revisiting Small Batch Training for Deep Neural Networks — HN submission

Fri, 27 Apr 2018 11:58:05 GMT

Revisiting Small Batch Training for Deep Neural Networks

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Revisiting Small Batch Training for Deep Neural Networks — HN submission

Thu, 26 Apr 2018 21:26:49 GMT

Revisiting Small Batch Training for Deep Neural Networks

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Revisiting Small Batch Training for Deep Neural Networks — HN submission

Mon, 23 Apr 2018 08:43:44 GMT

Revisiting Small Batch Training for Deep NNs; batch_size less than 64 is best

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.