DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients · HackerPapers discussions

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients — mentioned by WithinReason

Fri, 29 Mar 2024 17:05:42 GMT

1-bit weights have been a thing since at least 2016: https://arxiv.org/abs/1606.06160

Paper and discussion history. A mention is not an endorsement.

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients — mentioned by rfoo

Thu, 29 Feb 2024 08:09:33 GMT

This is wrong. The paper described that their activation is in int8 during inference. That being said, before-LLM-era deep learning already had low bit quantization down to 1w2f [0] working back in 2016 [1]. So it's certainly possible it would work for LLM too. [0] 1-bit weights, 2-bit activations; though practically people deployed 2w4f instead. [1] https://arxiv.org/abs/1606.06160

Paper and discussion history. A mention is not an endorsement.

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients — mentioned by paulsutter

Mon, 17 Apr 2017 19:20:30 GMT

There's nothing about the TPU architecture that prevents use for training. Training is also done as tensor operations, and training can be done at limited precision[1]. Just because the first TPU was memory-limited doesn't mean TPUs can never be used for training. [1] https://arxiv.org/abs/1606.06160

Paper and discussion history. A mention is not an endorsement.

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients — mentioned by mannigfaltig

Wed, 05 Apr 2017 22:57:04 GMT

The brain appears to spend about 4.7 bits per synapse (26 discernible states, given the noisy computation environment of the brain); so it seems to be plenty enough for general intelligence. This could, of course, merely be a biological limit and on silicon more fine-grained weights might be the optimum. Here is another paper demonstrating very good results with just 6 bit gradients: https://arxiv.org/abs/1606.06160

Paper and discussion history. A mention is not an endorsement.

DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients — mentioned by paulsutter

Sat, 11 Feb 2017 19:00:42 GMT

The article misses the core issue: neural network architectures are still in flux. "Neuromorphic" chips are hardwired to one architecture, which makes them power efficient but less flexible. When designs are more stable, such chips could be more practical. Meanwhile, upcoming (non-neuromorphic) AI processors are taking two directions: larger numbers of simplified GPU-type cores (such as NVIDIA Xavier and Intel's Lake Crest/Nervana chips), and FPGAs. Simplifying cores means lower precision, as fp32 and fp64 are overkill for neural networks and take up lots of silicon. The current NVIDIA Pascal…

Paper and discussion history. A mention is not an endorsement.