MetaFormer Is Actually What You Need for Vision · HackerPapers discussions

MetaFormer Is Actually What You Need for Vision — mentioned by slickytail

Thu, 22 May 2025 09:16:33 GMT

The relative unimportance of the exact SDPA attention in use in modern transformers is already known: https://arxiv.org/abs/2111.11418 The FFN, normalization, and residual connections are absolutely irreplaceable -- but attention can be replaced with almost any other layer that shares information between tokens, such as pooling, convolution, random mixing, etc.

Paper and discussion history. A mention is not an endorsement.

MetaFormer Is Actually What You Need for Vision — mentioned by whimsicalism

Sun, 23 Jul 2023 05:26:41 GMT

> 1) there are a lot of papers claiming to be the successor to the Transformer, and not all of them are cited; e.g., the MetaFormer is missing https://arxiv.org/abs/2111.11418. Another candidate that wasn't compares against (or at least argued why it wouldn't make sense to compare against) are the Hopfield Networks https://arxiv.org/abs/2008.02217. Neither of those papers are NLP applicable? And I think it's perfectly fair to focus on the alternatives (ie. like H3 and RWKV) that have been able to scale up to LLM levels and perplexity, which neither of the alternatives you mention have. Should…

Paper and discussion history. A mention is not an endorsement.

MetaFormer Is Actually What You Need for Vision — mentioned by sfriedr

Sun, 23 Jul 2023 04:55:58 GMT

I only spent a few minutes skimming thr paper, but: 1) there are a lot of papers claiming to be the successor to the Transformer, and not all of them are cited; e.g., the MetaFormer is missing https://arxiv.org/abs/2111.11418. Another candidate that wasn't compares against (or at least argued why it wouldn't make sense to compare against) are the Hopfield Networks https://arxiv.org/abs/2008.02217. So until a more solid Related Work section is written (their section is actually called "Relation to and Differences from Previous Methods") I reserve the right to be skeptical whether their model…

Paper and discussion history. A mention is not an endorsement.