HackerPapers

arXiv:2303.06349cs.LG

Resurrecting Recurrent Neural Networks for Long Sequences

Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, Soham De

Abstract

Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train. Deep state-space models (SSMs) have recently been shown to perform remarkably well on long sequence modeling tasks, and have the added benefits of fast parallelizable training and RNN-like fast inference. However, while SSMs are superficially similar to RNNs, there are important differences that make it unclear where their performance boost over RNNs comes from. In this paper, we show that careful design of deep RNNs using standard signal propagation arguments can recover the impressive performance of deep SSMs on long-range reasoning tasks, while also matching their training speed. To achieve this, we analyze and ablate a series of changes to standard RNNs including linearizing and diagonalizing the recurrence, using better parameterizations and initializations, and ensuring proper normalization of the forward pass. Our results provide new insights on the origins of the impressive performance of deep SSMs, while also introducing an RNN block called the Linear Recurrent Unit that matches both their performance on the Long Range Arena benchmark and their computational efficiency.

arXiv comments
30 pages, 9 figures
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 5 comments across 4 threads, from Apr 2023 to Mar 2025. Found in 1 HN submission from Mar 2023 to Mar 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Transformers Without Normalization 1 comment

    • Proper initialization of layers keeps gradient magnitudes from vanishing/exploding in deep networks. If you make sure the output of each layer has mean 0, std 1, the gradients will be reasonable as well, for example. I recommend e.g. the og resnet paper and its follow-up from…

      tsurbaView on HNParent context

  2. Were RNNs all we needed? 1 comment

    • Nonlinearity somewhere is fundamental, but it doesn't need to be between each layer. You can, for instance, project each input to a higher dimensional space with a nonlinearity, and the problem becomes linearly separable with high probability (cf Cover's Theorem). So, for XOR…

      scarmigView on HNParent context

  3. Scaling Transformers to 1B Tokens 2 comments

    • > The bitter lesson [1] is going to eventually come for all of these. Eventually we'll figure out how to machine-learn the heuristic rather than hard code it. Recurrent neural networks (RNNs) do this implicitly, but we don't yet know how to effectively train RNNs on ultra-deep…

      euclaiseView on HNParent context

    • > Recurrent neural networks (RNNs) do this implicitly, but we don't yet know how to effectively train RNNs on ultra-deep sequences. What would you call "ultra-deep"? [1] shows how to train an RNN in a GPT-like mode, using parallel scan, with great performance on Path-X, which…

      smaddoxView on HNParent context

  4. From deep to long learning? 1 comment

    • Other approaches for long sequence lengths: - "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf - "Resurrecting Recurrent Neural Networks for Long Sequences" https://arxiv.org/pdf/2303.06349.pdf It's unclear (at least to me) if…

      smaddoxView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.