What Algorithms can Transformers Learn? A Study in Length Generalization · HackerPapers discussions

What Algorithms can Transformers Learn? A Study in Length Generalization — mentioned by Vetch

Sat, 14 Dec 2024 15:09:48 GMT

!Long post warning! Tokenization is often scapegoated for many transformer limitations. I suppose it's because reading about the many limitations of the transformer architecture is harder than dumping everything on tokenization (which to be fair, is often indirectly involved with or exacerbating some deeper issue). > Why can't LLM spell words? Tokenization. LLMs can spell if you ask them to though. And there have been investigations into this capability (ref:2). Tokenization makes computations that involve spelling more difficult, but this is downstream of deeper computational limitations of…

Paper and discussion history. A mention is not an endorsement.

What Algorithms can Transformers Learn? A Study in Length Generalization — mentioned by hansonw

Sat, 27 Apr 2024 16:09:21 GMT

This is also a good paper on the subject: What Algorithms can Transformers Learn? A Study in Length Generalization https://arxiv.org/abs/2310.16028

Paper and discussion history. A mention is not an endorsement.

What Algorithms can Transformers Learn? A Study in Length Generalization — mentioned by astrange

Sun, 25 Feb 2024 05:09:52 GMT

We can prove that transformers can do computation beyond memorization, and that they can learn it from examples, unless you think learning an algorithm is memorization. https://arxiv.org/abs/2301.05217 https://arxiv.org/abs/2310.16028

Paper and discussion history. A mention is not an endorsement.

What Algorithms can Transformers Learn? A Study in Length Generalization — HN submission

Fri, 09 Feb 2024 02:55:05 GMT

What Algorithms Can Transformers Learn? A Study in Length Generalization

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.