Theoretical limitations of multi-layer Transformer · HackerPapers discussions

Theoretical limitations of multi-layer Transformer — mentioned by namaria

Wed, 07 May 2025 11:26:08 GMT

On Limitations of the Transformer Architecture https://arxiv.org/abs/2402.08164 Theoretical limitations of multi-layer Transformer https://arxiv.org/abs/2412.02975

Paper and discussion history. A mention is not an endorsement.

Theoretical limitations of multi-layer Transformer — mentioned by namaria

Mon, 21 Apr 2025 20:23:37 GMT

Theoretical limitations of multi-layer Transformer https://arxiv.org/abs/2412.02975 On Limitations of the Transformer Architecture https://arxiv.org/abs/2402.08164 Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory https://arxiv.org/abs/2405.16674 TL;DR transformers are inherently limited with tasks requiring composition of sequential steps

Paper and discussion history. A mention is not an endorsement.

Theoretical limitations of multi-layer Transformer — mentioned by svachalek

Thu, 03 Apr 2025 19:20:15 GMT

This is an interesting paper, it postulates that the ability of an LLM to perform tasks correlates mostly to the number of layers it has, and that reasoning creates virtual layers in the context space. https://arxiv.org/abs/2412.02975

Paper and discussion history. A mention is not an endorsement.

Theoretical limitations of multi-layer Transformer — mentioned by orbital-decay

Sun, 02 Feb 2025 06:26:29 GMT

It's Quanta being misleading. They mention several papers but end up with this [1] which talks about decoder-only transformers, not LLMs in general, chatbots, or LLM synthesis engines, whatever that means. The paper also proves that CoT-like planning lets you squeeze more computation from a transformer, which is... obvious? but formally proven this time. Models trained to do CoT don't have some magical on-the-fly compositional ability, they just invest more computation (could be dozens millions of tokens in case of o3 solving the tasks from that benchmark). [1] https://arxiv.org/abs/2412.02975

Paper and discussion history. A mention is not an endorsement.

Theoretical limitations of multi-layer Transformer — mentioned by orbital-decay

Sun, 02 Feb 2025 06:12:39 GMT

That's because it uses a long CoT. The actual paper [1] [2] talks about the limitations of decoder-only transformers predicting the reply directly, although it also establishes the benefits of CoT for composition. This is all known for a long time and makes intuitive sense - you can't squeeze more computation from it than it can provide. The authors just formally proved it (which is no small deal). And Quanta is being dramatic with conclusions and headlines, as always. [1] https://arxiv.org/abs/2412.02975 [2] https://news.ycombinator.com/item?id=42889786

Paper and discussion history. A mention is not an endorsement.

Theoretical limitations of multi-layer Transformer — HN submission

Fri, 31 Jan 2025 17:48:23 GMT

Theoretical limitations of multi-layer Transformer

107 points; 22 discussion replies on HN. Paper and source history. A submission is not an endorsement.