HackerPapers

arXiv:2310.16764cs.CV

ConvNets Match Vision Transformers at Scale

Samuel L. Smith, Andrew Brock, Leonard Berrada, Soham De

Abstract

Many researchers believe that ConvNets perform well on small or moderately sized datasets, but are not competitive with Vision Transformers when given access to datasets on the web-scale. We challenge this belief by evaluating a performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset of images often used for training foundation models. We consider pre-training compute budgets between 0.4k and 110k TPU-v4 core compute hours, and train a series of networks of increasing depth and width from the NFNet model family. We observe a log-log scaling law between held out loss and compute budget. After fine-tuning on ImageNet, NFNets match the reported performance of Vision Transformers with comparable compute budgets. Our strongest fine-tuned model achieves a Top-1 accuracy of 90.4%.

arXiv comments
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.LGMachine Learning cs.NENeural and Evolutionary Computing

On Hacker News

Linked by 4 identifiable commenters in 6 comments across 4 threads, from Mar 2024 to Dec 2025. Found in 3 HN submissions from Oct 2023 to Oct 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. The Illustrated Transformer 1 comment

    • No, not at all. There is a transformer obsession that is quite possibly not supported by the actual facts (CNNs can still do just as well: https://arxiv.org/abs/2310.16764), and CNNs definitely remain preferable for smaller and more specialized tasks (e.g. computer vision on…

      D-MachineView on HNParent context

  2. Automatic differentiation can be incorrect 1 comment

    • Yup, I am pretty up to date with the literature (mostly) and read about DenseNet a long time ago (and have trained trained and tuned quite a few of them too). I've also done enough experimentation with tuning custom CNN architectures (on smaller medical datasets, mind you) where…

      D-MachineView on HNParent context

  3. Vision Transformers Are Overrated 3 comments

    • Pulling out a key part of this post from a DeepMind 2023 paper[1]: “Although the success of ViTs in computer vision is extremely impressive, in our view there is no strong evidence to suggest that pre-trained ViTs outperform pre-trained ConvNets when evaluated fairly.” Another…

      rocaucView on HNParent context

    • I recommend checking out this paper: https://arxiv.org/pdf/2310.16764.pdf. From the concluding section: "Although the success of ViTs in computer vision is extremely impressive, in our view there is no strong evidence to suggest that pre-trained ViTs outperform pre-trained…

      fzliuView on HNParent context

    • > You're double-dipping into the data. You look at the performance, then you tune some part of the network by hand, see if it helps, and then keep doing that. It's testing on the training data. I purposely tried to avoid adding any niche network modifications that would help it…

      fzliuView on HNParent context

  4. The Google employees who created transformers 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.