HackerPapers

arXiv:1606.06160cs.NE

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients

Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, Yuheng Zou

Abstract

We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers. As convolutions during forward/backward passes can now operate on low bitwidth weights and activations/gradients respectively, DoReFa-Net can use bit convolution kernels to accelerate both training and inference. Moreover, as bit convolutions can be efficiently implemented on CPU, FPGA, ASIC and GPU, DoReFa-Net opens the way to accelerate training of low bitwidth neural network on these hardware. Our experiments on SVHN and ImageNet datasets prove that DoReFa-Net can achieve comparable prediction accuracy as 32-bit counterparts. For example, a DoReFa-Net derived from AlexNet that has 1-bit weights, 2-bit activations, can be trained from scratch using 6-bit gradients to get 46.1\% top-1 accuracy on ImageNet validation set. The DoReFa-Net AlexNet model is released publicly.

arXiv comments
Journal reference
DOI
Subjects
cs.NENeural and Evolutionary Computing cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 5 comments across 5 threads, from Feb 2017 to Mar 2024.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Towards 1-bit Machine Learning Models 1 comment
  2. The Era of 1-bit LLMs: ternary parameters for cost-effective computing 1 comment

    • This is wrong. The paper described that their activation is in int8 during inference. That being said, before-LLM-era deep learning already had low bit quantization down to 1w2f [0] working back in 2016 [1]. So it's certainly possible it would work for LLM too. [0] 1-bit…

      rfooView on HNParent context

  3. Intel Discontinues the Intel Developer Forum; IDF17 Cancelled 1 comment

    • There's nothing about the TPU architecture that prevents use for training. Training is also done as tensor operations, and training can be done at limited precision[1]. Just because the first TPU was memory-limited doesn't mean TPUs can never be used for training. [1]…

      paulsutterView on HNParent context

  4. An In-Depth Look at Google's Tensor Processing Unit Architecture 1 comment

    • The brain appears to spend about 4.7 bits per synapse (26 discernible states, given the noisy computation environment of the brain); so it seems to be plenty enough for general intelligence. This could, of course, merely be a biological limit and on silicon more fine-grained…

      mannigfaltigView on HNParent context

  5. Intel Gets Serious About Neuromorphic, Cognitive Computing Future 1 comment

    • The article misses the core issue: neural network architectures are still in flux. "Neuromorphic" chips are hardwired to one architecture, which makes them power efficient but less flexible. When designs are more stable, such chips could be more practical. Meanwhile, upcoming…

      paulsutterView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.