Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality · HackerPapers discussions

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — mentioned by cs702

Wed, 04 Feb 2026 00:28:22 GMT

> Who out there is working on ... infinite history? Many people are still working on improving RNNs, mostly in academia. Examples off the top of my head: * RWKV: https://arxiv.org/abs/2006.16236 / https://arxiv.org/abs/2404.05892 https://arxiv.org/abs/2305.13048 * Linear attention: https://arxiv.org/abs/2503.14456 * State space models: https://arxiv.org/abs/2312.00752 / https://arxiv.org/abs/2405.21060 * Linear RNNs: https://arxiv.org/abs/2410.01201 Industry OTOH has gone all-in on Transformers.

Paper and discussion history. A mention is not an endorsement.

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — mentioned by juank10

Tue, 17 Jun 2025 07:48:38 GMT

Funnily enough, the code was deleted in the repo, but can still be seen in the commits. It's what you would expect from the paper :D On the general topic of non-attention LLMs, I recommend checking out the MesaNet [1], Rodimus [2], Gated DeltaNet [3], or Mamba2 [4]. They are currently SOTA. However, I have yet to see a compelling non attention based model that achieves good performance on code, math, reasoning, or multi-turn QA tasks. I do not think we are getting rid of attention soon, I believe the ability to look back is crucial in certain tasks. [1] https://arxiv.org/abs/2506.05233 [2]…

Paper and discussion history. A mention is not an endorsement.

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — mentioned by knowaveragejoe

Mon, 28 Oct 2024 19:48:03 GMT

Your intuition is, I think, pretty close to accurate. See this paper from earlier this year: > While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. We show that these families of models are actually quite closely related, and develop a rich framework of theoretical connections between SSMs and variants of attention, connected through various decompositions of a well-studied class of structured semiseparable matrices.…

Paper and discussion history. A mention is not an endorsement.

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — mentioned by epistasis

Mon, 14 Oct 2024 23:10:14 GMT

Tri Gao and Albert Gu say "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" https://arxiv.org/abs/2405.21060 Mamba-2 is used in Zamab2.

Paper and discussion history. A mention is not an endorsement.

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — HN submission

Tue, 04 Jun 2024 04:37:47 GMT

Transformers Are SSMs: Generalized Models and Efficient Algorithms Through

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality — HN submission

Mon, 03 Jun 2024 09:25:56 GMT

Transformers are SSMs (Mamba-2)

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.