Theoretical Limitations of Self-Attention in Neural Sequence Models · HackerPapers discussionshttps://hackerpaper.kaipfister.de/p/1906.06755/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMTTheoretical Limitations of Self-Attention in Neural Sequence Models — mentioned by dcolkitthttps://news.ycombinator.com/item?id=25654757urn:hackerpapers:mention:1906.06755:25654757Wed, 06 Jan 2021 04:51:35 GMT<p>The root-case of skepticism has always been that while Transformers do exceptionally well on finite-sized tasks, they lack any fully recursive understanding of the concepts.[0] A human can learn basic arithmetic, then generalize those principles to bigger number arithmetic, then go from there to algebra, then calculus, then so. Successively building on previously learned concepts in a fully recursive manner. Transformers are limited by the exponential size of their network. So GPT-3 does very well with 2-digit addition and okay with 2-digit multiplication, but can&#x27;t abstract to 6-digit…</p><p><a href="https://hackerpaper.kaipfister.de/p/1906.06755/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.CL