Were RNNs All We Needed? · HackerPapers discussions

Were RNNs All We Needed? — mentioned by cs702

Wed, 04 Feb 2026 00:28:22 GMT

> Who out there is working on ... infinite history? Many people are still working on improving RNNs, mostly in academia. Examples off the top of my head: * RWKV: https://arxiv.org/abs/2006.16236 / https://arxiv.org/abs/2404.05892 https://arxiv.org/abs/2305.13048 * Linear attention: https://arxiv.org/abs/2503.14456 * State space models: https://arxiv.org/abs/2312.00752 / https://arxiv.org/abs/2405.21060 * Linear RNNs: https://arxiv.org/abs/2410.01201 Industry OTOH has gone all-in on Transformers.

Paper and discussion history. A mention is not an endorsement.

Were RNNs All We Needed? — mentioned by akst

Sun, 21 Sep 2025 03:40:01 GMT

I think the author may have linked to the wrong paper in their opening paragraph (hopefully they see this and update the link) https://arxiv.org/abs/2410.01201 I opened it wondering what RNNs were, and for anyone else wondering they are apparently Recurrent Neural Networks, though you find that answer if you continue reading on (though the lack of a definition kind of stopped me in my tracks).

Paper and discussion history. A mention is not an endorsement.

Were RNNs All We Needed? — mentioned by rsfern

Tue, 01 Jul 2025 03:50:12 GMT

This paper “were RNNs all we needed?” explores this hypothesis a bit, finding that some pre-transformer sequence models can match transformers when trained at appropriate scale. Though they did have to make some modifications to unlock more parallelism https://arxiv.org/abs/2410.01201

Paper and discussion history. A mention is not an endorsement.

Were RNNs All We Needed? — mentioned by inciampati

Mon, 17 Mar 2025 17:57:01 GMT

An example, which is interesting, in which "deep" networks are necessary, is discussed in this fascinating and popular recent paper on RNNs [1]. Despite the fact that the minGRU and minLSTM models they propose don't explicitly model ordered state dependency, they can learn them as long as they are deep enough (deep >= 3): > Instead of explicitly modelling dependencies on previous states to capture long-range dependencies, these kinds of recurrent models can learn them by stacking multiple layers. [1] https://arxiv.org/abs/2410.01201

Paper and discussion history. A mention is not an endorsement.

Were RNNs All We Needed? — mentioned by bravura

Thu, 02 Jan 2025 17:15:44 GMT

Recent work by Feng et al from Bengio's lab focus on how attention can be formulated as an RNN ("Attention as RNN": https://arxiv.org/pdf/2405.13956) and how minimal versions of GRUs and LSTMs can be trained in parallel by removing some parameters ("Were RNNs All We Needed?" https://arxiv.org/pdf/2410.01201). It's possible we start seeing more blended version of RNN/attention architecture exploring different LLM properties. In particular, Aaren architecture in the former paper "can not only (i) be trained in parallel (like Transformers) but also (ii) be updated efficiently with new tokens…

Paper and discussion history. A mention is not an endorsement.

Were RNNs All We Needed? — mentioned by cs702

Sat, 14 Dec 2024 21:23:59 GMT

Thanks for the quick reply! Interesting. I would have thought one of those "minimum viable" RNNs (like https://arxiv.org/abs/2410.01201) would have been ideal for this. I might tinker a bit with this :-)

Paper and discussion history. A mention is not an endorsement.

Were RNNs All We Needed? — mentioned by xnx

Thu, 03 Oct 2024 18:25:23 GMT

It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning) Curve-fitting is about embedding a dataset on a curve. The critical factor is the dataset, not the…

Paper and discussion history. A mention is not an endorsement.

Were RNNs All We Needed? — HN submission

Thu, 03 Oct 2024 17:31:59 GMT

Were RNNs all we needed?

520 points; 260 discussion replies on HN. Paper and source history. A submission is not an endorsement.