Linear Transformers Are Secretly Fast Weight Programmers · HackerPapers discussions

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by logicchains

Tue, 22 Sep 2026 07:00:17 GMT

>Also obligatory: Schmidhuber talked about this a long time ago. And again recently: https://arxiv.org/abs/2102.11174

Paper and discussion history. A mention is not an endorsement.

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by korbip

Sat, 15 Feb 2025 03:04:16 GMT

There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t}…

Paper and discussion history. A mention is not an endorsement.

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by crotchfire

Thu, 22 Feb 2024 09:14:59 GMT

Actually it is related. Transformers are just networks that learn to program the weights of other networks [1]. In the successful cases the programmed network has been quite primitive -- merely a key-value store -- in order to ensure that you can backpropagate errors from the programmed network's outputs all the way to the programmer network's inputs. The present work extends this idea to a different kind of programmed network: a convolutional image-processing network. There are many more breakthroughs to be achieved along this line of research -- it is a rich vein to mine. I believe our best…

Paper and discussion history. A mention is not an endorsement.

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by KirillPanov

Thu, 05 Oct 2023 10:12:17 GMT

Personally I think the key/value/query terminology is kind of crappy, but at this point we're stuck with it. Think of the "fast" network as being like a hashmap. It stores key-value pairs. You feed it a query, which is a key that you're looking for, and it gives you back the corresponding value. Unlike the hashmaps you're used to, however, this is sort of a "blurry hashmap". If you ask for a key you'll get sort of a blend of the values nearby it -- even if the key isn't in the hashmap. > Are the key value pairs a vector and the last layer an actual token in a simple LLM, for example? Almost.…

Paper and discussion history. A mention is not an endorsement.

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by KirillPanov

Thu, 05 Oct 2023 09:41:17 GMT

> misses the query/key/value weight Did you click the right link? The words "query", "key", and "value" are in the image! For the rest, you'll want to read the paper: https://arxiv.org/abs/2102.11174 Embeddings were around long before transformers. The image only depicts a single attention head, of course.

Paper and discussion history. A mention is not an endorsement.

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by treyd

Wed, 04 Oct 2023 21:23:50 GMT

You may be interested in "Linear Transformers Are Secretly Fast Weight Programmers": https://arxiv.org/abs/2102.11174

Paper and discussion history. A mention is not an endorsement.

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by ntonozzi

Fri, 22 Sep 2023 22:24:03 GMT

Yes! Check out this paper describing how Linear Transformers are secretly Fast Weight Programmers: https://arxiv.org/abs/2102.11174.

Paper and discussion history. A mention is not an endorsement.

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by albertzeyer

Sun, 03 Sep 2023 11:50:58 GMT

Those equivalences can connect two different fields and allows to transfer methods from one field to the other. Each field usually has developed quite a number of methods and tricks over the time. So when this work shows that they are equivalent (with restrictions), you can maybe use some of the tricks of SVMs and try to use them to improve the Transformer model or its training. Otherwise, they just help us in better understanding Transformers and SVMs. There have been similar equivalences before, for example: Linear Transformers Are Secretly Fast Weight Programmers…

Paper and discussion history. A mention is not an endorsement.

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by cma

Mon, 01 May 2023 14:37:45 GMT

> The authors of Attention is all you need, the transformers paper, y Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before: https://arxiv.org/abs/2102.11174

Paper and discussion history. A mention is not an endorsement.

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by cma

Mon, 23 Jan 2023 20:49:35 GMT

Fast weight programmers: https://arxiv.org/abs/2102.11174

Paper and discussion history. A mention is not an endorsement.

Linear Transformers Are Secretly Fast Weight Programmers — mentioned by Skyy93

Tue, 25 Oct 2022 08:37:41 GMT

As you should know: Schmidhuber even invented the Transformer-Models, to be read in: https://arxiv.org/abs/2102.11174 ;)

Paper and discussion history. A mention is not an endorsement.