Neighborhood Attention Transformer — mentioned by godelski

Sun, 09 Apr 2023 21:32:46 GMT

It is also important to remember that there are different inductive biases for different networks. Sutton's "Bitter Lesson"[0] argues for scale + more arbitrary complexity. ViTs scale really well because some of this arbitrary complexity. But I agree with your point that what to use is not obvious and this gets really complicated real quickly. For example, Swin loses some of the properties of self-attention but Neighborhood Attention doesn't[1] (these are both considered reductive attention types). Does this have a large effect? Might depend on your task. Looking at non-classification tasks…

Paper and discussion history. A mention is not an endorsement.

Neighborhood Attention Transformer · HackerPapers discussions

Neighborhood Attention Transformer — mentioned by godelski