An Attention Free Transformer · HackerPapers discussions

An Attention Free Transformer — mentioned by nl

Fri, 20 Jan 2023 00:16:15 GMT

This isn't true - the model architecture matters a lot. In general RNNs cannot handle long term dependencies (ie, long pieces of text) because the gradient vanishes. It's unclear how this solves this problem although they do reference the "attention free transformer" paper: https://arxiv.org/abs/2105.14103

Paper and discussion history. A mention is not an endorsement.

An Attention Free Transformer — mentioned by nl

Thu, 19 Jan 2023 22:40:35 GMT

For those wondering how on earth they are getting decent results from a RNN without long range forgetting, I don't really know either! But they reference https://arxiv.org/abs/2105.14103 and the bottom section of https://github.com/BlinkDL/RWKV-LM has an explainer.

Paper and discussion history. A mention is not an endorsement.