HackerPapers

arXiv:2102.11174cs.LG

Linear Transformers Are Secretly Fast Weight Programmers

Imanol Schlag, Kazuki Irie, Jürgen Schmidhuber

Abstract

We show the formal equivalence of linearised self-attention mechanisms and fast weight controllers from the early '90s, where a ``slow" neural net learns by gradient descent to program the ``fast weights" of another net through sequences of elementary programming instructions which are additive outer products of self-invented activation patterns (today called keys and values). Such Fast Weight Programmers (FWPs) learn to manipulate the contents of a finite memory and dynamically interact with it. We infer a memory capacity limitation of recent linearised softmax attention variants, and replace the purely additive outer products by a delta rule-like programming instruction, such that the FWP can more easily learn to correct the current mapping from keys to values. The FWP also learns to compute dynamically changing learning rates. We also propose a new kernel function to linearise attention which balances simplicity and effectiveness. We conduct experiments on synthetic retrieval problems as well as standard machine translation and language modelling tasks which demonstrate the benefits of our methods.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 9 identifiable commenters in 11 comments across 10 threads, from Oct 2022 to Sep 2026.

Comments linking this paper, per year.

Discussed in comments10 threads

  1. Transformers Explained Visually 1 comment
  2. Ask HN: Is anybody building an alternative transformer? 1 comment

    • There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15]…

      korbipView on HNParent context

  3. Neural Network Diffusion 1 comment

    • Actually it is related. Transformers are just networks that learn to program the weights of other networks [1]. In the successful cases the programmed network has been quite primitive -- merely a key-value store -- in order to ensure that you can backpropagate errors from the…

      crotchfireView on HNParent context

  4. How Transformers Work 2 comments

    • > misses the query/key/value weight Did you click the right link? The words "query", "key", and "value" are in the image! For the rest, you'll want to read the paper: https://arxiv.org/abs/2102.11174 Embeddings were around long before transformers. The image only depicts a…

      KirillPanovView on HNParent context

    • Personally I think the key/value/query terminology is kind of crappy, but at this point we're stuck with it. Think of the "fast" network as being like a hashmap. It stores key-value pairs. You feed it a query, which is a key that you're looking for, and it gives you back the…

      KirillPanovView on HNParent context

  5. Ring attention with blockwise transformers for near-infinite context 1 comment

    • You may be interested in "Linear Transformers Are Secretly Fast Weight Programmers": https://arxiv.org/abs/2102.11174

      treydView on HNParent context

  6. I made a transformer to predict a simple sequence manually 1 comment

    • Yes! Check out this paper describing how Linear Transformers are secretly Fast Weight Programmers: https://arxiv.org/abs/2102.11174.

      ntonozziView on HNParent context

  7. Transformers as Support Vector Machines 1 comment

    • Those equivalences can connect two different fields and allows to transfer methods from one field to the other. Each field usually has developed quite a number of methods and tricks over the time. So when this work shows that they are equivalent (with restrictions), you can…

      albertzeyerView on HNParent context

  8. Geoffrey Hinton leaves Google and warns of danger ahead 1 comment

    • > The authors of Attention is all you need, the transformers paper, y Schmidhuber claims to have invented something formally equivalent to the linear Transformer architecture (slightly weaker) years before: https://arxiv.org/abs/2102.11174

      cmaView on HNParent context

  9. Deepmind is working on a rival to ChatGPT 1 comment
  10. Meta Learning Backpropagation and Improving It (2020) 1 comment

    • As you should know: Schmidhuber even invented the Transformer-Models, to be read in: https://arxiv.org/abs/2102.11174 ;)

      Skyy93View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.