Neural Arithmetic Logic Units · HackerPapers discussions

Neural Arithmetic Logic Units — mentioned by williamtrask

Sat, 13 Apr 2024 17:15:53 GMT

Fwiw - I wrote a paper documenting LLM (and neural) regression a few years ago (Neurips 2018) https://arxiv.org/abs/1808.00508 They can interpolate but extrapolation is hard because of the constraints from nonlinearities

Paper and discussion history. A mention is not an endorsement.

Neural Arithmetic Logic Units — mentioned by williamtrask

Fri, 11 Feb 2022 10:02:41 GMT

Poor performance is more likely due to how transformer neural networks view numbers. It memorises them like words instead of modeling their numerical structure. Thus even if it’s seen the number 3456 and 3458, it knows nothing of 3457. Totally different embedding. It’s like a kid memorising a multiplication table instead of learning the more general principle of multiplication (related: this illusion is why big models are so popular. Memorise more stuff.) Paper (NeurIPS/DeepMind): https://arxiv.org/abs/1808.00508

Paper and discussion history. A mention is not an endorsement.

Neural Arithmetic Logic Units — HN submission

Fri, 03 Aug 2018 09:11:31 GMT

Neural Arithmetic Logic Units – Learning Numbers & Arithmetic End-To-End

7 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.