Overcoming a Theoretical Limitation of Self-Attention — mentioned by shawntan

Tue, 18 Apr 2023 15:09:15 GMT

> I thought the key takeaway is that nonlinearity is important. Multilayer perceptrons can be collapsed into an equivalent single layer if there isn't nonlinearity thrown in. multi-input XOR can be solved using a form of sin(x) function after summing in a single layer (sin squared if you don't like negative numbers). Some element of this is used in this paper: https://arxiv.org/pdf/2202.12172.pdf More broadly, as I've alluded to in other replies, XOR is not the problem in itself, but illustrates a bigger problem. Yes, non-linearities can solve problems, but one hidden layer and the wrong…

Paper and discussion history. A mention is not an endorsement.

Overcoming a Theoretical Limitation of Self-Attention · HackerPapers discussions

Overcoming a Theoretical Limitation of Self-Attention — mentioned by shawntan