Efficient Time-Series Approximation with Linear Recurrent Neural Networks: Architecture Learning and Predictive Power — mentioned by scarmig

Fri, 04 Oct 2024 12:42:15 GMT

Nonlinearity somewhere is fundamental, but it doesn't need to be between each layer. You can, for instance, project each input to a higher dimensional space with a nonlinearity, and the problem becomes linearly separable with high probability (cf Cover's Theorem). So, for XOR, (x, y) -> (x, y, xy), and it becomes trivial for a linear NN to solve. Architectures like Mamba have a linear recurrent state space system as their core, so even though you need a nonlinearity somewhere, it doesn't need to be pervasive. And linear recurrent networks are surprisingly powerful…

Paper and discussion history. A mention is not an endorsement.

Efficient Time-Series Approximation with Linear Recurrent Neural Networks: Architecture Learning and Predictive Power · HackerPapers discussions

Efficient Time-Series Approximation with Linear Recurrent Neural Networks: Architecture Learning and Predictive Power — mentioned by scarmig