Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet · HackerPapers discussions

Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet — mentioned by hprotagonist

Fri, 10 Feb 2023 19:13:24 GMT

https://arxiv.org/abs/2105.02723 The strong performance of vision transformers on image classification and other vision tasks is often attributed to the design of their multi-head attention layers. However, the extent to which attention is responsible for this strong performance remains unclear. In this short report, we ask: is the attention layer even necessary? Specifically, we replace the attention layer in a vision transformer with a feed-forward layer applied over the patch dimension. The resulting architecture is simply a series of feed-forward layers applied over the patch and feature…

Paper and discussion history. A mention is not an endorsement.

Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet — HN submission

Wed, 18 Jan 2023 00:00:18 GMT

A stack of feed-forward layers does surprisingly well on ImageNet

97 points; 25 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet — HN submission

Wed, 19 May 2021 08:28:02 GMT

ML: Do You Even Need Attention? A Stack of FF Layers Does Surprisingly Well

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.