An Empirical Model of Large-Batch Training · HackerPapers discussions

An Empirical Model of Large-Batch Training — mentioned by deddy

Tue, 31 Dec 2024 03:47:44 GMT

Great question! There are a few different aspects to this. With gradient agreement filtering having a greater number of batches (generally) increases the likelihood of finding another microbatch that agrees with the gradient simply by virtue of having more gradient "samples" to compare. So having more batches increases the chance of success there. The algorithm as laid out in the paper is a simple approach to combining groups of batches where larger numbers batches doesn't necessarily improve you chances of success if the batch you're comparing against is an outlier itself. There are almost…

Paper and discussion history. A mention is not an endorsement.

An Empirical Model of Large-Batch Training — HN submission

Thu, 07 Nov 2024 23:35:16 GMT

An Empirical Model of Large-Batch Training (2018)

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

An Empirical Model of Large-Batch Training — HN submission

Tue, 14 May 2024 16:48:56 GMT

An Empirical Model of Large-Batch Training

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.