Deep neural networks are robust to weight binarization and other non-linear distortions · HackerPapers discussions

Deep neural networks are robust to weight binarization and other non-linear distortions — mentioned by fc417fc802

Fri, 20 Jun 2025 03:33:46 GMT

It's been done. [0][1] But the fact that BitNet [2] is possible calls into question the need for it. NNs are surprisingly robust to a wide array of distortions. [3] [0] https://arxiv.org/abs/1603.01025 [1] https://par.nsf.gov/servlets/purl/10163070 [2] https://arxiv.org/abs/2310.11453 [3] https://arxiv.org/abs/1606.01981

Paper and discussion history. A mention is not an endorsement.

Deep neural networks are robust to weight binarization and other non-linear distortions — mentioned by paul_mk1

Fri, 29 Mar 2024 17:57:23 GMT

Sub 1-bit has been done at least as far back as 2016 for VGG style networks (my work). I was able to get 0.68 "effective" bits. The idea is that in each forward pass you add noise to each weight independently drawn from normal distribution, and when you calculate snr, it's sub 1 bit. Points to the idea that a stochastic memory element can be used. https://arxiv.org/abs/1606.01981

Paper and discussion history. A mention is not an endorsement.

Deep neural networks are robust to weight binarization and other non-linear distortions — mentioned by paul_mk1

Wed, 28 Feb 2024 22:48:52 GMT

Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there). Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work). Helpful hint to authors: you can get down to 0.68 bits / weight using a similar technique, good chance this will work for LLMs too. https://arxiv.org/abs/1606.01981 This was a passion project of mine in my last few months at IBM research :). I am convinced there is a deep…

Paper and discussion history. A mention is not an endorsement.

Deep neural networks are robust to weight binarization and other non-linear distortions — HN submission

Thu, 09 Jun 2016 08:13:30 GMT

Deep neural networks are robust to nonlinear weight distortions

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.