HackerPapers

arXiv:2405.21060cs.LG

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Tri Dao, Albert Gu

Abstract

While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. We show that these families of models are actually quite closely related, and develop a rich framework of theoretical connections between SSMs and variants of attention, connected through various decompositions of a well-studied class of structured semiseparable matrices. Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.

arXiv comments
ICML 2024
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 4 comments across 4 threads, from Oct 2024 to Feb 2026. Found in 2 HN submissions from Jun 2024 to Jun 2024. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. FlashAttention-T: Towards Tensorized Attention 1 comment

    • > Who out there is working on ... infinite history? Many people are still working on improving RNNs, mostly in academia. Examples off the top of my head: * RWKV: https://arxiv.org/abs/2006.16236 / https://arxiv.org/abs/2404.05892 https://arxiv.org/abs/2305.13048 * Linear…

      cs702View on HNParent context

  2. Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons 1 comment

    • Funnily enough, the code was deleted in the repo, but can still be seen in the commits. It's what you would expect from the paper :D On the general topic of non-attention LLMs, I recommend checking out the MesaNet [1], Rodimus [2], Gated DeltaNet [3], or Mamba2 [4]. They are…

      juank10View on HNParent context

  3. State-space models can learn in-context by gradient descent 1 comment

    • Your intuition is, I think, pretty close to accurate. See this paper from earlier this year: > While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or…

      knowaveragejoeView on HNParent context

  4. Zamba2-7B 1 comment

    • Tri Gao and Albert Gu say "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" https://arxiv.org/abs/2405.21060 Mamba-2 is used in Zamab2.

      epistasisView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.