HackerPapers

arXiv:1502.02551cs.LG

Deep Learning with Limited Numerical Precision

Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, Pritish Narayanan

Abstract

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.

arXiv comments
10 pages, 6 figures, 1 table
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.NENeural and Evolutionary Computing stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 5 threads, from May 2015 to Apr 2017. Found in 1 HN submission from May 2018 to May 2018. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. An In-Depth Look at Google's Tensor Processing Unit Architecture 2 comments

    • Here's a paper they published a little while ago about limited numerical precision and deep learning: https://arxiv.org/abs/1502.02551

      shepardrtcView on HNParent context

    • There are hints that even less than 8 bits per weight might be usable (for certain cases and on custom hw). Not sure if it's practical but it is definitely interesting. https://arxiv.org/pdf/1502.02551.pdf https://arxiv.org/pdf/1610.00324.pdf I wanted to have some basic idea…

      koja86View on HNParent context

  2. The rise of AI is creating new variety in the chip market, and trouble for Intel 1 comment

    • For deep learning or chips for deep learning? With an EE/Chem/Math background you should be set to go :) For deep learning, I highly recommend the cs231n course materials (available for free online) and the Deep Learning Book by Goodfellow et al. For chip design for deep…

      deepnotderpView on HNParent context

  3. TensorFlow: open-source library for machine intelligence 1 comment

    • There are a few people that seem to be working with integer-based deep learning and FPGAs, and even though they're at the beginning of research, things are looking pretty promising. Here's a paper that was released earlier this year that seems to show that at least some…

      SanddancerView on HNParent context

  4. Why Are Eight Bits Enough for Deep Neural Networks? 1 comment

    • The problem with using digital calculations is that they are deterministic. If a result is really small, it is just rounded down to zero. So if you add a bunch of small numbers, you get zero. Even if the result should be large. Stochastic rounding can fix this. You round each…

      HoushalterView on HNParent context

  5. Fast Approximate Logarithms, Part I: The Basics 1 comment

    • Indeed, 64 bits of accuracy is overkill for a lot of ML algorithms, where there is so much noise that the additional quantization noise due to low precision is negligible. Most deep neural nets are already being trained in single precision because that is what NVIDIA GPUs can do…

      benanneView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.