Teaching Arithmetic to Small Transformers · HackerPapers discussions

Teaching Arithmetic to Small Transformers — mentioned by famouswaffles

Mon, 30 Sep 2024 23:47:38 GMT

> there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers. fundamental inability ? No. Current Sota LLM (4o, claude, gemini) woes with arithmetic is not a transformer weakness never mind a large language modelling one. Those benchmarks show that those particular models have problems with accuracy on that many digits, not that LLMs fundamentally cannot be accurate on that many digits. https://arxiv.org/abs/2405.17399 https://arxiv.org/abs/2307.03381 https://arxiv.org/abs/2310.02989 Ultimately, numbers are…

Paper and discussion history. A mention is not an endorsement.

Teaching Arithmetic to Small Transformers — mentioned by gwern

Tue, 28 May 2024 15:01:31 GMT

Yes. This has already been demonstrated by "Teaching Arithmetic to Small Transformers" https://arxiv.org/abs/2307.03381 , I'm not sure what OP adds except demonstrating that you can do that via the embedding itself rather than the tokenization. > We first demonstrate that conventional training data is not the most effective for arithmetic learning, and simple formatting changes can significantly improve accuracy. This leads to sharp phase transitions as a function of training data scale, which, in some cases, can be explained through connections to low-rank matrix completion. Building on…

Paper and discussion history. A mention is not an endorsement.

Teaching Arithmetic to Small Transformers — HN submission

Thu, 13 Jul 2023 18:38:44 GMT

Teaching Arithmetic to Small Transformers

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Teaching Arithmetic to Small Transformers — HN submission

Mon, 10 Jul 2023 15:44:29 GMT

Teaching Arithmetic to Small Transformers

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.