Scalable Methods for 8-bit Training of Neural Networks — mentioned by CuriouslyC

Mon, 06 Mar 2023 03:23:00 GMT

It can go farther than that, it seems like the weight gradients are the main thing where the precision is a bottleneck (see https://arxiv.org/abs/1805.11046).

Paper and discussion history. A mention is not an endorsement.

Scalable Methods for 8-bit Training of Neural Networks — mentioned by p1esk

Wed, 08 May 2019 23:46:19 GMT

Thank you for the detailed answer. I think your main point is that memory bandwidth would prevent the performance speedup. Are V100s memory bound when executing F16 ops on tensor cores? Second, do we really need dedicated FP32 cores for DL? Tensor cores accumulate in FP32 (is that what you meant when you said they did a significant amount of FP32 compute?), and recent papers indicate we’re moving towards 8 bit training [1]. Besides, do TPUs use dedicated FP32 hw? Finally, if the memory bandwidth is indeed the bottleneck, perhaps all that die area from FP32 and especially FP64 cores could be…

Paper and discussion history. A mention is not an endorsement.

Scalable Methods for 8-bit Training of Neural Networks · HackerPapers discussions

Scalable Methods for 8-bit Training of Neural Networks — mentioned by CuriouslyC

Scalable Methods for 8-bit Training of Neural Networks — mentioned by p1esk