On Limitations of the Transformer Architecture · HackerPapers discussions

On Limitations of the Transformer Architecture — mentioned by namaria

Wed, 07 May 2025 11:26:08 GMT

On Limitations of the Transformer Architecture https://arxiv.org/abs/2402.08164 Theoretical limitations of multi-layer Transformer https://arxiv.org/abs/2412.02975

Paper and discussion history. A mention is not an endorsement.

On Limitations of the Transformer Architecture — mentioned by namaria

Mon, 21 Apr 2025 20:23:37 GMT

Theoretical limitations of multi-layer Transformer https://arxiv.org/abs/2412.02975 On Limitations of the Transformer Architecture https://arxiv.org/abs/2402.08164 Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory https://arxiv.org/abs/2405.16674 TL;DR transformers are inherently limited with tasks requiring composition of sequential steps

Paper and discussion history. A mention is not an endorsement.

On Limitations of the Transformer Architecture — mentioned by JohnKemeny

Sun, 19 May 2024 12:08:31 GMT

On Limitations of the Transformer Architecture https://arxiv.org/abs/2402.08164

Paper and discussion history. A mention is not an endorsement.

On Limitations of the Transformer Architecture — mentioned by JohnKemeny

Mon, 08 Apr 2024 07:23:00 GMT

There's a paper by Papadimitriou (from Logicomix fame) and some collaborators that the transformer model is incapable of solving certain simple problems, and if done by Cost, it needs exponentially many steps. The paper is currently only available at Arxiv (ie not yet peer-reviewed), but given that it is Papadimitriou, I would be inclined to believe the results. https://arxiv.org/abs/2402.08164

Paper and discussion history. A mention is not an endorsement.