AdderNet: Do We Really Need Multiplications in Deep Learning? — mentioned by fgfm

Wed, 28 Feb 2024 18:12:10 GMT

It's funny how discoveries in NLP & computer vision complement each other. The replacement of multiplication by additions made me think about the AdderNet paper (https://arxiv.org/abs/1912.13200), which concluded as you had to suffer almost no performance drop. Perhaps the accumulators in current hardware cannot leverage this to its full potential, but combined with such a strict quantization, this would open LLM to the wider ML community much earlier than expected (when consumer hardware allows you to train near SOTA LLMs from scratch on your machine).

Paper and discussion history. A mention is not an endorsement.

AdderNet: Do We Really Need Multiplications in Deep Learning? · HackerPapers discussions

AdderNet: Do We Really Need Multiplications in Deep Learning? — mentioned by fgfm