Training deep neural networks with low precision multiplications — mentioned by Houshalter

Thu, 15 Sep 2016 10:51:57 GMT

Backpropagation can work with any precision, as long as you use stochastic rounding (so that the rounding errors are not correlated.) Without stochastic rounding even 16 bits will have rounding error bias. http://arxiv.org/abs/1412.7024

Paper and discussion history. A mention is not an endorsement.

Training deep neural networks with low precision multiplications — mentioned by benanne

Sat, 09 May 2015 18:08:56 GMT

Indeed, 64 bits of accuracy is overkill for a lot of ML algorithms, where there is so much noise that the additional quantization noise due to low precision is negligible. Most deep neural nets are already being trained in single precision because that is what NVIDIA GPUs can do the fastest. There has been quite some research on reducing the bit depth further the last couple of years, see e.g. http://arxiv.org/abs/1502.02551 and http://arxiv.org/abs/1412.7024. The current generation of GPUs already has limited support for half-precision operations, and the tools for using these operations in…

Paper and discussion history. A mention is not an endorsement.

Training deep neural networks with low precision multiplications · HackerPapers discussions

Training deep neural networks with low precision multiplications — mentioned by Houshalter

Training deep neural networks with low precision multiplications — mentioned by benanne