ResMLP: Feedforward networks for image classification with data-efficient training — mentioned by godelski

Wed, 29 Nov 2023 19:35:50 GMT

> Also, we know that transformers can scale Do we have strong evidence that other models don't scale or have we just put more time into transformers? Convolutional resnets look to scale on vision and language: (cv) https://arxiv.org/abs/2301.00808, (cv) https://arxiv.org/abs/2110.00476, (nlp) https://github.com/HazyResearch/safari MLPs also seem to scale: (cv) https://arxiv.org/abs/2105.01601, (cv) https://arxiv.org/abs/2105.03404 I mean I don't see a strong reason to turn away from attention as well but I also don't think anyone's thrown a billion parameter MLP or Conv model at a problem.…

Paper and discussion history. A mention is not an endorsement.

ResMLP: Feedforward networks for image classification with data-efficient training · HackerPapers discussions

ResMLP: Feedforward networks for image classification with data-efficient training — mentioned by godelski