An argument in favor of strong scaling for deep neural networks with small datasets — mentioned by kvathupo

Thu, 01 Sep 2022 04:03:29 GMT

Although this seems to only affect the Chinese and Russian governments (as opposed to researchers), I'm generally against economic brinkmanship of this variety. It just festers resentment, and doesn't seem to achieve much: of what little I could find and scan in an hour, it appears that Nvidia GPUs strong-scale pretty well for neural networks ([1], [2]). Although this depends heavily on the NN architecture and training algorithm, Ampere architectures _seem_ to suffer from similar levels of atomic contention as Volta architectures [3]. That said, the throughput for half-precision floating…

Paper and discussion history. A mention is not an endorsement.

An argument in favor of strong scaling for deep neural networks with small datasets · HackerPapers discussions

An argument in favor of strong scaling for deep neural networks with small datasets — mentioned by kvathupo