Deep Learning with Limited Numerical Precision · HackerPapers discussions

Deep Learning with Limited Numerical Precision — HN submission

Sun, 20 May 2018 10:05:18 GMT

Deep Learning with Limited Numerical Precision

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Deep Learning with Limited Numerical Precision — mentioned by koja86

Thu, 06 Apr 2017 09:03:59 GMT

There are hints that even less than 8 bits per weight might be usable (for certain cases and on custom hw). Not sure if it's practical but it is definitely interesting. https://arxiv.org/pdf/1502.02551.pdf https://arxiv.org/pdf/1610.00324.pdf I wanted to have some basic idea about hardware so I did some "research" (googling) and ended up giving a short informal talk. My slides with some links are here: http://ml.kjx.cz/hw4nn2017.pdf

Paper and discussion history. A mention is not an endorsement.

Deep Learning with Limited Numerical Precision — mentioned by shepardrtc

Wed, 05 Apr 2017 18:45:17 GMT

Here's a paper they published a little while ago about limited numerical precision and deep learning: https://arxiv.org/abs/1502.02551

Paper and discussion history. A mention is not an endorsement.

Deep Learning with Limited Numerical Precision — mentioned by deepnotderp

Mon, 27 Feb 2017 01:24:43 GMT

For deep learning or chips for deep learning? With an EE/Chem/Math background you should be set to go :) For deep learning, I highly recommend the cs231n course materials (available for free online) and the Deep Learning Book by Goodfellow et al. For chip design for deep learning specifically, it's a fairly new field, and a lot of commercial interests, hence why a lot of it isn't available in the form of an "Intro to Deep Learning Chip Design" course. But, the basic point is that deep learning can both train and perform inference in astonishingly low precision. For training…

Paper and discussion history. A mention is not an endorsement.

Deep Learning with Limited Numerical Precision — mentioned by Sanddancer

Mon, 09 Nov 2015 23:34:50 GMT

There are a few people that seem to be working with integer-based deep learning and FPGAs, and even though they're at the beginning of research, things are looking pretty promising. Here's a paper that was released earlier this year that seems to show that at least some algorithms can see a big benefit from FPGAs http://arxiv.org/pdf/1502.02551v1.pdf .

Paper and discussion history. A mention is not an endorsement.

Deep Learning with Limited Numerical Precision — mentioned by Houshalter

Sun, 20 Sep 2015 00:26:44 GMT

The problem with using digital calculations is that they are deterministic. If a result is really small, it is just rounded down to zero. So if you add a bunch of small numbers, you get zero. Even if the result should be large. Stochastic rounding can fix this. You round each step with the probability so it's expected value is the same. Usually it will round down to 0, but sometimes it will round up to 1. Relevant paper, using stochastic rounding. Without it the results get worse and worse before you even get to 8 bits. With stochastic rounding, there is no performance degradation. You could…

Paper and discussion history. A mention is not an endorsement.

Deep Learning with Limited Numerical Precision — mentioned by benanne

Sat, 09 May 2015 18:08:56 GMT

Indeed, 64 bits of accuracy is overkill for a lot of ML algorithms, where there is so much noise that the additional quantization noise due to low precision is negligible. Most deep neural nets are already being trained in single precision because that is what NVIDIA GPUs can do the fastest. There has been quite some research on reducing the bit depth further the last couple of years, see e.g. http://arxiv.org/abs/1502.02551 and http://arxiv.org/abs/1412.7024. The current generation of GPUs already has limited support for half-precision operations, and the tools for using these operations in…

Paper and discussion history. A mention is not an endorsement.