MLP-Mixer: An all-MLP Architecture for Vision · HackerPapers discussions

MLP-Mixer: An all-MLP Architecture for Vision — mentioned by mattalex

Fri, 05 Jun 2026 17:19:17 GMT

Because the size of the attention matrix depends on the number of tokens (this is what makes attention N^2). If you don't care about having a flexible number of input tokens (e.g. in image processing) you can learn a fixed routing matrix. This is known as an MLP mixer https://arxiv.org/pdf/2105.01601 : you have one layer that processes each token in isolation ("vertical MLP") but ignores the inter-token connections, followed by a layer that combines between tokens ("horizontal MLP") that treats the internals of every token identically.

Paper and discussion history. A mention is not an endorsement.

MLP-Mixer: An all-MLP Architecture for Vision — mentioned by godelski

Wed, 29 Nov 2023 19:35:50 GMT

> Also, we know that transformers can scale Do we have strong evidence that other models don't scale or have we just put more time into transformers? Convolutional resnets look to scale on vision and language: (cv) https://arxiv.org/abs/2301.00808, (cv) https://arxiv.org/abs/2110.00476, (nlp) https://github.com/HazyResearch/safari MLPs also seem to scale: (cv) https://arxiv.org/abs/2105.01601, (cv) https://arxiv.org/abs/2105.03404 I mean I don't see a strong reason to turn away from attention as well but I also don't think anyone's thrown a billion parameter MLP or Conv model at a problem.…

Paper and discussion history. A mention is not an endorsement.

MLP-Mixer: An all-MLP Architecture for Vision — HN submission

Sun, 19 Sep 2021 03:57:55 GMT

MLP-Mixer: An All-MLP Architecture for Vision

38 points; 16 discussion replies on HN. Paper and source history. A submission is not an endorsement.

MLP-Mixer: An all-MLP Architecture for Vision — mentioned by gwern

Sun, 09 May 2021 02:08:45 GMT

Actually... we're not sure! While it's true that Transformers work amazingly on more domains than they have any right to, it's unclear if they truly obsolete multi-layer perceptrons/fully-connected networks. You may have seen the recent splash of MLP-Mixer for image classification using perceptrons (https://arxiv.org/abs/2105.01601#google) but FC-heavy or FC-only nets have popped up here and there, never quite going away, and slowly getting better initializations & residual layers making them ever more trainable. I have a bibliography of some links I've noted over the years about them…

Paper and discussion history. A mention is not an endorsement.

MLP-Mixer: An all-MLP Architecture for Vision — HN submission

Wed, 05 May 2021 15:05:13 GMT

MLP-Mixer: An All-MLP Architecture for Vision

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.