Resurrecting Recurrent Neural Networks for Long Sequences · HackerPapers discussions

Resurrecting Recurrent Neural Networks for Long Sequences — mentioned by tsurba

Sat, 15 Mar 2025 11:27:06 GMT

Proper initialization of layers keeps gradient magnitudes from vanishing/exploding in deep networks. If you make sure the output of each layer has mean 0, std 1, the gradients will be reasonable as well, for example. I recommend e.g. the og resnet paper and its follow-up from Kaiming He et al. For a modern take on RNNs, read https://arxiv.org/abs/2303.06349 by DeepMind. There essentially the point is that largest eigenvalue (spectral radius) needs to be around 1, meaning repeated applications of a linear transformation doesn’t cause increase or decrease of the activations.

Paper and discussion history. A mention is not an endorsement.

Resurrecting Recurrent Neural Networks for Long Sequences — mentioned by scarmig

Fri, 04 Oct 2024 12:42:15 GMT

Nonlinearity somewhere is fundamental, but it doesn't need to be between each layer. You can, for instance, project each input to a higher dimensional space with a nonlinearity, and the problem becomes linearly separable with high probability (cf Cover's Theorem). So, for XOR, (x, y) -> (x, y, xy), and it becomes trivial for a linear NN to solve. Architectures like Mamba have a linear recurrent state space system as their core, so even though you need a nonlinearity somewhere, it doesn't need to be pervasive. And linear recurrent networks are surprisingly powerful…

Paper and discussion history. A mention is not an endorsement.

Resurrecting Recurrent Neural Networks for Long Sequences — mentioned by smaddox

Fri, 07 Jul 2023 01:27:25 GMT

> Recurrent neural networks (RNNs) do this implicitly, but we don't yet know how to effectively train RNNs on ultra-deep sequences. What would you call "ultra-deep"? [1] shows how to train an RNN in a GPT-like mode, using parallel scan, with great performance on Path-X, which has a sequence length of 16k. It's based on prior papers doing the same thing but from a state space model perspective. [1] https://arxiv.org/abs/2303.06349

Paper and discussion history. A mention is not an endorsement.

Resurrecting Recurrent Neural Networks for Long Sequences — mentioned by euclaise

Thu, 06 Jul 2023 15:20:05 GMT

> The bitter lesson [1] is going to eventually come for all of these. Eventually we'll figure out how to machine-learn the heuristic rather than hard code it. Recurrent neural networks (RNNs) do this implicitly, but we don't yet know how to effectively train RNNs on ultra-deep sequences. Linear RNNs and RWKV are examples of RNNs on deep sequences: https://arxiv.org/abs/2303.06349 https://arxiv.org/abs/2305.13048

Paper and discussion history. A mention is not an endorsement.

Resurrecting Recurrent Neural Networks for Long Sequences — mentioned by smaddox

Mon, 10 Apr 2023 17:09:42 GMT

Other approaches for long sequence lengths: - "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf - "Resurrecting Recurrent Neural Networks for Long Sequences" https://arxiv.org/pdf/2303.06349.pdf It's unclear (at least to me) if any of these scale as well as transformers at this point.

Paper and discussion history. A mention is not an endorsement.

Resurrecting Recurrent Neural Networks for Long Sequences — HN submission

Tue, 14 Mar 2023 01:45:43 GMT

Resurrecting Recurrent Neural Networks for Long Sequences

4 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.