Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention · HackerPapers discussions

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by fheinsen

Wed, 04 Feb 2026 17:58:12 GMT

[3] is linear attention, https://arxiv.org/abs/2006.16236, a well-known result with ~3K citations: https://scholar.google.com/scholar_lookup?arxiv_id=2006.1623...

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by fheinsen

Wed, 04 Feb 2026 17:41:45 GMT

This is a form of linear attention (https://arxiv.org/abs/2006.16236) that approximates standard scaled dot-product attention to arbitrary precision, by adding Taylor terms in an efficient manner. Each additional Taylor term improves the approximation. Efficiency is achieved by exploiting certain mathematical symmetries that become evident only after decomposing the standard formulation of attention into an expression over chains of tensor products. The github repository's README walks through examples. The first example is with 8 Taylor terms.

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by cs702

Wed, 04 Feb 2026 00:28:22 GMT

> Who out there is working on ... infinite history? Many people are still working on improving RNNs, mostly in academia. Examples off the top of my head: * RWKV: https://arxiv.org/abs/2006.16236 / https://arxiv.org/abs/2404.05892 https://arxiv.org/abs/2305.13048 * Linear attention: https://arxiv.org/abs/2503.14456 * State space models: https://arxiv.org/abs/2312.00752 / https://arxiv.org/abs/2405.21060 * Linear RNNs: https://arxiv.org/abs/2410.01201 Industry OTOH has gone all-in on Transformers.

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by korbip

Sat, 15 Feb 2025 03:04:16 GMT

There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t}…

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by tripplyons

Fri, 14 Feb 2025 17:14:22 GMT

Here is a paper explaining it: https://arxiv.org/abs/2006.16236

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by SomewhatLikely

Sat, 31 Aug 2024 06:31:52 GMT

https://arxiv.org/abs/2006.16236 Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by Fripplebubby

Mon, 01 Jul 2024 16:38:15 GMT

To further problematize this question (which I don't feel like I can actually answer), consider this paper: "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention" - https://arxiv.org/pdf/2006.16236 What this shows is that actually a specific narrow definition of transformer (a transformer with "causal masking" - see paper) is equivalent to an RNN, and vice versa. Similarly Mamba (https://arxiv.org/abs/2312.00752), the other hot architecture at the moment, has an equivalent unit to a gated RNN. For performance reasons, I believe they use an equivalent CNN during…

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by cs702

Mon, 03 Jun 2024 17:06:53 GMT

TL;DR: The authors show that if you simplify Mamba so its state-space layer uses a diagonal matrix A that is a scalar times the identity matrix, the state-space transformation can be expressed as a form of causal linear attention.[a] That's the duality the authors refer to in the title. The key practical benefit is that it enables more efficient (faster) training on GPUs. --- [a] https://arxiv.org/abs/2006.16236

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by cs702

Mon, 03 Jun 2024 16:57:32 GMT

Dao and Gu show that if you simplify Mamba so its state-space layer uses a diagonal matrix A_t that is a scalar times the identity matrix, i.e., A_t = a_t I, the state-space transformation can be expressed as a form of causal linear attention[a] by compounding coefficients a_1 ... a_t at each time step t. The equivalence of the simplified state-space layer and causal linear attention constitute the duality the authors refer to in the title. By taking advantage of this duality, Mamba-2 can be trained more efficiently, i.e., faster than original Mamba on GPUs. Theoretical stuff aside, Mamba-2's…

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by cs702

Tue, 05 Dec 2023 12:26:56 GMT

This looks great. Thank you Gu and Dao! For those who are unaware, previous "state-space models" (SSMs) took the classic state space model from control theory[a], parameterized its matrices, and applied it in discrete time to tokens in a sequence. Many people, including me, got excited by SSMs, given their linear cost... but they failed to live up to the hype. The key difference now is that, instead of parameterizing the transformation matrices as before, the authors propose computing them from the data. In other words, instead of applying the same transformation at each step in time (which…

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by MacsHeadroom

Sat, 22 Apr 2023 19:51:04 GMT

Transformers are turing complete: https://arxiv.org/abs/1901.03429 Additionally, Transformers are RNNs: https://arxiv.org/abs/2006.16236 Transformers are turing complete RNNs, as are the Neocortex. There are differences like spiking vs non-spiking neurons, universal vs non-universal activation, the effects of brain waves, bicameralism, etc. But these differences may or may not be important depending on the topic of discussion. Additionally, the differences increasingly disappear as transformers are increasingly augmented with functionally equivalent systems.

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by GistNoesis

Sat, 22 Apr 2023 16:43:18 GMT

^^ See above comment and "Transformers are RNNs" paper to convince yourself. There are various ways of seeing how transformer architectures work. For the past data, using a temporal causal mask, you can compute all past features as they would have been seen by applying the causal mask, which allows you to seemingly do all the past computations in parallel, which hide the sequential aspect of it. What I've described correspond to the the simpler subvariant figure 1 of "Attention is all you need" https://arxiv.org/pdf/2006.16236.pdf , of the transformer architecture in the case where you don't…

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by GistNoesis

Sat, 22 Apr 2023 13:22:41 GMT

> They don’t have any internal state This is a common misconception : they do have internal state. Transformers are RNNs : section 3.4 (page 5) https://arxiv.org/pdf/2006.16236.pdf

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by PartiallyTyped

Fri, 20 Jan 2023 10:20:23 GMT

The key component is the linear attention[1] and residual connections. [1] https://arxiv.org/abs/2006.16236 > Transformers achieve remarkable performance in several tasks but due to their quadratic complexity, with respect to the input's length, they are prohibitively slow for very long sequences. To address this limitation, we express the self-attention as a linear dot-product of kernel feature maps and make use of the associativity property of matrix products to reduce the complexity from (N2) to (N), where N is the sequence length. We show that this formulation permits an iterative…

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by GistNoesis

Thu, 19 Jan 2023 22:59:01 GMT

>As there is no hidden state, no "memory" or "intention" can be passed to the future This is probably your misconception : "Transformers are RNNs" https://arxiv.org/pdf/2006.16236.pdf (see section 3.4) Even though an inner state is not specified explicitly like in a LSTM, for a transformer it's possible to view the inner state as implicitly defined as the inner features (deterministically derived from the layers weights) applied to the past inputs. Therefore transformers can learn to do the task by learning to remember a,b,ab to this inner state and write them in order ab,a,b (Though to train…

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — mentioned by 101101001010

Fri, 03 Jul 2020 12:45:46 GMT

They are also efficient at inference-time. On GPUs the difference is noticeable only for sequences of length > 1024 (2048 for reformer since it adds some operations for hashing) thanks to the massive parallelism of GPUs amortizing the quadratic effect of the "usual" self-attention mechanism. [edit] Linformer (https://arxiv.org/pdf/2006.04768.pdf) is a different project from the one linked in https://linear-transformers.com/ (Transformers are RNNs https://arxiv.org/pdf/2006.16236.pdf).

Paper and discussion history. A mention is not an endorsement.

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention — HN submission

Fri, 03 Jul 2020 07:55:10 GMT

Faster Autoregressive Transformers with Linear Attention

6 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.