Investigating the Limitations of Transformers with Simple Arithmetic Tasks · HackerPapers discussions

Investigating the Limitations of Transformers with Simple Arithmetic Tasks — mentioned by spyder

Thu, 30 Mar 2023 21:24:10 GMT

That's also an interesting one, but I think I found it, but it's a little different from what I remembered because it's not just the scientific notation that helps but more so the addition of positional tokens: https://arxiv.org/abs/2102.13019 The essence of it from the abstract: ". In particular, the model fails to learn addition of five-digit numbers when using subwords (e.g., “32”), and it struggles to learn with character-level representations (e.g., “3 2”). By introducing position tokens (e.g., “3 10e1 2”), the model learns to accurately add and subtract numbers up to 60 digits. We…

Paper and discussion history. A mention is not an endorsement.

Investigating the Limitations of Transformers with Simple Arithmetic Tasks — mentioned by versteegen

Mon, 06 Feb 2023 04:46:34 GMT

I wrote that double descent is widespread, not grokking. Of course a transformer can't do multiplication or any other kind of operation on an infinite set of numbers, because it has only bounded depth which limits the number of steps it can emulate of any algorithm. But I think I see how I could build a transformer by hand that could multiply any two 4-digit numbers. The difficulty is the quadratic number of steps. Addition and subtraction are far easier, [1] shows that can be solved: "By introducing position tokens (e.g., "3 10e1 2"), the model learns to accurately add and subtract numbers…

Paper and discussion history. A mention is not an endorsement.