BinaryConnect: Training Deep Neural Networks with binary weights during propagations · HackerPapers discussions

BinaryConnect: Training Deep Neural Networks with binary weights during propagations — mentioned by p1esk

Sat, 09 Nov 2024 18:14:28 GMT

People did care back then. This paper had jumpstarted the whole model compression field (which used to be a hot area of research in early 90s): https://arxiv.org/abs/1511.00363 Before that, in 2012, Alexnet had to be partially split into two submodels, running on two GPUs (using a form of interlayer grouped convolutions) because it could not fit in 3GB of a single 580 card. Ternary networks appeared in 2016. Unless you mean you actually tried to train in ternary precision - clearly not possible with any gradient based optimization methods.

Paper and discussion history. A mention is not an endorsement.

BinaryConnect: Training Deep Neural Networks with binary weights during propagations — mentioned by rapatel0

Fri, 29 Mar 2024 17:33:20 GMT

Don't forget Michael https://arxiv.org/abs/1511.00363 < 2015

Paper and discussion history. A mention is not an endorsement.

BinaryConnect: Training Deep Neural Networks with binary weights during propagations — mentioned by paul_mk1

Sat, 09 Mar 2024 22:22:55 GMT

Goes back before then. This got popularized by BinaryConnect in 2015, and groups were training binary networks as early as 2011. You are probably referring to XNOR net, and the novel piece there was also using binary activations (which bitnet is not). So as far as I can tell, bitnet is basically BinaryConnect applied to LLMs. https://arxiv.org/abs/1511.00363

Paper and discussion history. A mention is not an endorsement.

BinaryConnect: Training Deep Neural Networks with binary weights during propagations — mentioned by fmap

Wed, 09 Mar 2016 14:57:45 GMT

There is a lot of room for improvement with the implementation. The way we are using deep neural networks at the moment is exellent for prototyping, but far from optimal. For instance, this paper http://arxiv.org/abs/1511.00363 shows that you can replace floating point operations by simple bitwise operations without losing too much precision in DNNs for image recognition. Together with a better (that is, compiled, instead of interpreted) representation of the inference step I would expect an order of magnitude improvement at a small loss in precision. More software tuning, especially the kind…

Paper and discussion history. A mention is not an endorsement.

BinaryConnect: Training Deep Neural Networks with binary weights during propagations — HN submission

Thu, 11 Feb 2016 20:29:17 GMT

Training Deep Neural Networks with binary weights during propagations

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.