Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — mentioned by neilmovva

Mon, 12 Feb 2018 21:49:38 GMT

It matters when defining parallel work distribution. Unless memory bandwidth is homogeneous across the whole board (i.e. each TPU on a board gets 600 GB/s to its peers), we can't do model parallelism across ASICs efficiently, and must fall back to data parallelism. Which is fine, until you run into limits on maximum batchsize (e.g. up to 8192, as FAIR was able to manage [1] with some tweaks to SGD). [1] https://arxiv.org/abs/1706.02677

Paper and discussion history. A mention is not an endorsement.

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour · HackerPapers discussions

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour — mentioned by neilmovva