Attention as an RNN · HackerPapers discussions

Attention as an RNN — mentioned by bravura

Fri, 14 Feb 2025 20:58:42 GMT

Check out "Attention as an RNN" by Feng et al (2024), with Bengio as a co-author. https://arxiv.org/pdf/2405.13956 Abstract: The advent of Transformers marked a significant breakthrough in sequence modelling, providing a highly performant architecture capable of leveraging GPU parallelism. However, Transformers are computationally expensive at inference time, limiting their applications, particularly in low-resource settings (e.g., mobile and embedded devices). Addressing this, we (1) begin by showing that attention can be viewed as a special Recurrent Neural Network (RNN) with the ability to…

Paper and discussion history. A mention is not an endorsement.

Attention as an RNN — mentioned by bravura

Thu, 02 Jan 2025 17:15:44 GMT

Recent work by Feng et al from Bengio's lab focus on how attention can be formulated as an RNN ("Attention as RNN": https://arxiv.org/pdf/2405.13956) and how minimal versions of GRUs and LSTMs can be trained in parallel by removing some parameters ("Were RNNs All We Needed?" https://arxiv.org/pdf/2410.01201). It's possible we start seeing more blended version of RNN/attention architecture exploring different LLM properties. In particular, Aaren architecture in the former paper "can not only (i) be trained in parallel (like Transformers) but also (ii) be updated efficiently with new tokens…

Paper and discussion history. A mention is not an endorsement.

Attention as an RNN — HN submission

Sun, 26 May 2024 11:32:24 GMT

Attention as an RNN

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.