The Evolved Transformer — mentioned by gwern

Sun, 03 Feb 2019 22:07:10 GMT

Self-attention uses 1x1 convolutions, and while the original Transformer is fully-connected-only (I think?) the latest & greatest uses convolutions (https://arxiv.org/abs/1901.11117), and even if you're using the original, it's very often being used in conjunction with convolutions elsewhere. So you could argue that the real title would have to be 'The Unreasonable Effectiveness of Attention', but given all the other stuff convolutions have done without attention like WaveNet, I think it's currently fairer to go with convolutions than attention. But we'll see how it goes over the next 4…

Paper and discussion history. A mention is not an endorsement.

The Evolved Transformer — HN submission

Fri, 01 Feb 2019 04:40:35 GMT

The Evolved Transformer

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

The Evolved Transformer · HackerPapers discussions

The Evolved Transformer — mentioned by gwern

The Evolved Transformer — HN submission