HackerPapers

arXiv:2402.01032cs.LG

Repeat After Me: Transformers are Better than State Space Models at Copying

Samy Jelassi, David Brandfonbrener, Sham M. Kakade, Eran Malach

Abstract

Transformers are the dominant architecture for sequence modeling, but there is growing interest in models that use a fixed-size latent state that does not depend on the sequence length, which we refer to as "generalized state space models" (GSSMs). In this paper we show that while GSSMs are promising in terms of inference-time efficiency, they are limited compared to transformer models on tasks that require copying from the input context. We start with a theoretical analysis of the simple task of string copying and prove that a two layer transformer can copy strings of exponential length while GSSMs are fundamentally limited by their fixed-size latent state. Empirically, we find that transformers outperform GSSMs in terms of efficiency and generalization on synthetic tasks that require copying the context. Finally, we evaluate pretrained large language models and find that transformer models dramatically outperform state space models at copying and retrieving information from context. Taken together, these results suggest a fundamental gap between transformers and GSSMs on tasks of practical interest.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.AIArtificial Intelligence cs.CLComputation and Language

On Hacker News

Linked by 4 identifiable commenters in 6 comments across 5 threads, from Feb 2024 to Oct 2024.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Were RNNs all we needed? 1 comment

    • My feeling is that the answer is "no", in the sense that these RNNs wouldn't be able to universally replace Transformers in LLMs, even though they might be good enough in some cases and beat them in others. Here's why. A user of an LLM might give the model some long text and…

      trottView on HNParent context

  2. Mamba-2 – State Space Duality 1 comment

    • Dao and Gu show that if you simplify Mamba so its state-space layer uses a diagonal matrix A_t that is a scalar times the identity matrix, i.e., A_t = a_t I, the state-space transformation can be expressed as a form of causal linear attention[a] by compounding coefficients a_1…

      cs702View on HNParent context

  3. Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length 1 comment

    • The two issues I see with this work: * There's no mention of model performance on recall tasks. Models with attention do well on recall tasks, but models without it, like this one, tend to do poorly.[a] What is the performance of this model on recall tasks? * As others here have…

      cs702View on HNParent context

  4. Visualizing Attention, a Transformer's Heart [video] 1 comment

    • Fantastic work by Grant Sanderson, as usual. Attention has won.[a] It deserves to be more widely understood. --- [a] Nothing has outperformed attention so far, not even Mamba: https://arxiv.org/abs/2402.01032

      cs702View on HNParent context

  5. Mamba: The Easy Way 2 comments

    • Too new is definitely one thing. Someone is going to have to make a gamble to actually paying for a serious pretraining run with this architecture before we know how it really stacks up against transformers. There are some papers suggesting that transformers are better than SSMs…

      thatguysaguyView on HNParent context

    • Indeed: https://arxiv.org/pdf/2402.01032.pdf Perhaps future iterations of SSMs will accommodate dynamically sized (but still non-linearly-growing) hidden states / memories!

      hansonwView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.