Deep Neural Network Training without Multiplications — mentioned by nickpsecurity

Tue, 28 May 2024 23:16:32 GMT

There's work on replacing multiplication. Here's four examples: https://openaccess.thecvf.com/content_CVPR_2020/papers/Chen_... https://arxiv.org/abs/2012.03458 https://openaccess.thecvf.com/content/CVPR2021W/MAI/papers/E... https://arxiv.org/pdf/2106.10860

Paper and discussion history. A mention is not an endorsement.

Deep Neural Network Training without Multiplications — mentioned by ffast-math

Wed, 01 Sep 2021 04:50:45 GMT

There's been some work on doing adds instead of multiplies (e.g., https://arxiv.org/abs/2012.03458). And I think float8 will roughly be doing this under the hood. Personally, I'm not sure whether this is the path to go down or not. Doing everything in log space could help from a hardware perspective, since multiply-adds are much more complex than adds. But it 1) doesn't reduce the number of operations being performed, 2) might increase the necessary bit width for a given level of fidelity (depending on your input distribution and other factors), and 3) might make accumulate operations…

Paper and discussion history. A mention is not an endorsement.

Deep Neural Network Training without Multiplications · HackerPapers discussions

Deep Neural Network Training without Multiplications — mentioned by nickpsecurity

Deep Neural Network Training without Multiplications — mentioned by ffast-math