HackerPapers

arXiv:1602.02830cs.LG

Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, Yoshua Bengio

Abstract

We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At training-time the binary weights and activations are used for computing the parameters gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs we conduct two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.

arXiv comments
11 pages and 3 figures
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 8 identifiable commenters in 8 comments across 8 threads, from Sep 2016 to Mar 2024.

Comments linking this paper, per year.

Discussed in comments8 threads

  1. Towards 1-bit Machine Learning Models 1 comment
  2. The Era of 1-bit LLMs: ternary parameters for cost-effective computing 1 comment

    • Prior art: Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1 https://arxiv.org/abs/1602.02830 Ternary Neural Networks for Resource-Efficient AI Applications https://arxiv.org/abs/1609.00222

      oxxoxoxoooView on HNParent context

  3. Deep Learning and Its Implications for Computer Architecture and Chip Design 1 comment

    • > i guess brain's synaptic precision could go way lower, as low as 26 distinct synapse weights: https://elifesciences.org/articles/10778 Thanks for the link. Artificial neural networks all the way down to binary weights [1] although this approach doesn't seem like the most…

      quotemstrView on HNParent context

  4. MIT AGI: Conversation with Yoshua Bengio [video] 1 comment
  5. Nonlinear Computation in Deep Linear Networks 1 comment
  6. Intel Gets Serious About Neuromorphic, Cognitive Computing Future 1 comment

    • The reality is that TrueNorth can use CNNs and be trained with backprop to achieve good performance with networks like ImageNet despite LeCun's comments [0, 1] LeCun's primary critique is that binary wont work; "to get good results on a task like ImageNet you need about 8 bit of…

      emcqView on HNParent context

  7. Intel will add deep-learning instructions to its processors 1 comment
  8. Stacked Approximated Regression Machine: A Simple Deep Learning Approach 1 comment

    • As far as I understand this, these guys claim they can train convolutional and many other types of deep neural nets faster by pretraining each layer with a new unsupervised technique via which the layer sort of learns to compress its inputs (a local optimization problem), and…

      cs702View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.