The Illusion of State in State-Space Models · HackerPapers discussions

The Illusion of State in State-Space Models — mentioned by shawntan

Thu, 02 Jan 2025 19:04:23 GMT

The formulations in attention as rnn have similar issues as rwkv. Fundamentally it's a question of what we call an RNN. Personally I think it's important not to call some of these recent architectures RNNs because they have theoretical properties that do not match (read: they're worse) what we've "classically" called RNNs. Ref: https://arxiv.org/abs/2404.08819 As a rule of thumb: you generally don't get parallelism for free, you pay for it with poorer expressivity.

Paper and discussion history. A mention is not an endorsement.

The Illusion of State in State-Space Models — mentioned by logicchains

Thu, 03 Oct 2024 18:57:27 GMT

The model in the paper isn't a "real" RNN due making it parallelizable, for same the reasons described in https://arxiv.org/abs/2404.08819 , and hence is theoretically less powerful than a "real" RNN (struggles at some classes of problems that RNNs traditionally excel at). On the other hand, https://arxiv.org/abs/2405.04517 contains a "real" RNN component, which demonstrates a significant improvement on the kind of state-tracking problems that transformers struggle with.

Paper and discussion history. A mention is not an endorsement.

The Illusion of State in State-Space Models — HN submission

Fri, 07 Jun 2024 11:53:27 GMT

The illusion of state in state-space models

60 points; 59 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Illusion of State in State-Space Models — mentioned by logicchains

Mon, 03 Jun 2024 19:50:42 GMT

Quadratic transformers outperform weaker forms of attention on recall tasks, but recurrent models are strictly more powerful than transformers (when they don't use chain of thought) at state tracking problems. Mamba 2 likely has the same limitations with state tracking as a full transformer due to being parallelizable: https://arxiv.org/abs/2404.08819 .

Paper and discussion history. A mention is not an endorsement.

The Illusion of State in State-Space Models — mentioned by opprobium

Wed, 22 May 2024 16:53:17 GMT

Recent example of a proof regarding theoretical limitations of Transformers: https://aclanthology.org/2023.tacl-1.31.pdf (also extended to cover SSMs https://arxiv.org/pdf/2404.08819)

Paper and discussion history. A mention is not an endorsement.

The Illusion of State in State-Space Models — mentioned by lucidrains

Wed, 08 May 2024 14:37:58 GMT

https://arxiv.org/abs/2404.08819

Paper and discussion history. A mention is not an endorsement.

The Illusion of State in State-Space Models — HN submission

Sun, 21 Apr 2024 19:06:20 GMT

The Illusion of State in State-Space Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Illusion of State in State-Space Models — HN submission

Wed, 17 Apr 2024 01:29:14 GMT

The illusion of state in state-space models

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.