Repeat After Me: Transformers are Better than State Space Models at Copying · HackerPapers discussions

Repeat After Me: Transformers are Better than State Space Models at Copying — mentioned by trott

Thu, 03 Oct 2024 20:36:27 GMT

My feeling is that the answer is "no", in the sense that these RNNs wouldn't be able to universally replace Transformers in LLMs, even though they might be good enough in some cases and beat them in others. Here's why. A user of an LLM might give the model some long text and then say "Translate this into German please". A Transformer can look back at its whole history. But what is an RNN to do? While the length of its context is unlimited, the amount of information the model retains about it is bounded by whatever is in its hidden state at any given time. Relevant…

Paper and discussion history. A mention is not an endorsement.

Repeat After Me: Transformers are Better than State Space Models at Copying — mentioned by cs702

Mon, 03 Jun 2024 16:57:32 GMT

Dao and Gu show that if you simplify Mamba so its state-space layer uses a diagonal matrix A_t that is a scalar times the identity matrix, i.e., A_t = a_t I, the state-space transformation can be expressed as a form of causal linear attention[a] by compounding coefficients a_1 ... a_t at each time step t. The equivalence of the simplified state-space layer and causal linear attention constitute the duality the authors refer to in the title. By taking advantage of this duality, Mamba-2 can be trained more efficiently, i.e., faster than original Mamba on GPUs. Theoretical stuff aside, Mamba-2's…

Paper and discussion history. A mention is not an endorsement.

Repeat After Me: Transformers are Better than State Space Models at Copying — mentioned by cs702

Tue, 16 Apr 2024 19:20:16 GMT

The two issues I see with this work: * There's no mention of model performance on recall tasks. Models with attention do well on recall tasks, but models without it, like this one, tend to do poorly.[a] What is the performance of this model on recall tasks? * As others here have pointed out, the github link is dead. In addition, the pretrained weights don't seem to be available on HF or anywhere else. It's hard to check any claims if we have neither code nor weights! --- [a] https://arxiv.org/abs/2402.01032

Paper and discussion history. A mention is not an endorsement.

Repeat After Me: Transformers are Better than State Space Models at Copying — mentioned by cs702

Mon, 15 Apr 2024 18:19:02 GMT

Fantastic work by Grant Sanderson, as usual. Attention has won.[a] It deserves to be more widely understood. --- [a] Nothing has outperformed attention so far, not even Mamba: https://arxiv.org/abs/2402.01032

Paper and discussion history. A mention is not an endorsement.

Repeat After Me: Transformers are Better than State Space Models at Copying — mentioned by hansonw

Sat, 24 Feb 2024 08:51:21 GMT

Indeed: https://arxiv.org/pdf/2402.01032.pdf Perhaps future iterations of SSMs will accommodate dynamically sized (but still non-linearly-growing) hidden states / memories!

Paper and discussion history. A mention is not an endorsement.

Repeat After Me: Transformers are Better than State Space Models at Copying — mentioned by thatguysaguy

Fri, 23 Feb 2024 17:47:24 GMT

Too new is definitely one thing. Someone is going to have to make a gamble to actually paying for a serious pretraining run with this architecture before we know how it really stacks up against transformers. There are some papers suggesting that transformers are better than SSMs in fundamental ways (e.g. They cannot do arbitrary key-based recall from their context: https://arxiv.org/abs/2402.01032). This means it's not just a no-brainer to switch over.

Paper and discussion history. A mention is not an endorsement.