ConvNets Match Vision Transformers at Scale · HackerPapers discussions

ConvNets Match Vision Transformers at Scale — mentioned by D-Machine

Tue, 23 Dec 2025 03:53:32 GMT

No, not at all. There is a transformer obsession that is quite possibly not supported by the actual facts (CNNs can still do just as well: https://arxiv.org/abs/2310.16764), and CNNs definitely remain preferable for smaller and more specialized tasks (e.g. computer vision on medical data). If you also get into more robust and/or specialized tasks (e.g. rotation invariant computer vision models, graph neural networks, models working on point-cloud data, etc) then transformers are also not obviously the right choice at all (or even usable in the first place). So plenty of other useful…

Paper and discussion history. A mention is not an endorsement.

ConvNets Match Vision Transformers at Scale — mentioned by D-Machine

Fri, 19 Sep 2025 02:11:28 GMT

Yup, I am pretty up to date with the literature (mostly) and read about DenseNet a long time ago (and have trained trained and tuned quite a few of them too). I've also done enough experimentation with tuning custom CNN architectures (on smaller medical datasets, mind you) where the inclusion of residual connections was a hyperparameter to be tuned. It is pretty obvious that with deeper networks, turning off the skip connections just makes your loss curves hit a premature (and bad) plateau far more often than not. Residual connections in CNNs clearly don't change the representational…

Paper and discussion history. A mention is not an endorsement.

ConvNets Match Vision Transformers at Scale — mentioned by fzliu

Tue, 02 Apr 2024 02:45:44 GMT

> You're double-dipping into the data. You look at the performance, then you tune some part of the network by hand, see if it helps, and then keep doing that. It's testing on the training data. I purposely tried to avoid adding any niche network modifications that would help it overfit to in-1k. All three of the modifications are applicable to other networks and datasets. > No one cares about ImageNet-1k. No one needs to classify ImageNet-1k in real life. I completely agree with you; I just don't have the compute to train this on a massive dataset. With that being said, I'm not advocating for…

Paper and discussion history. A mention is not an endorsement.

ConvNets Match Vision Transformers at Scale — mentioned by fzliu

Tue, 02 Apr 2024 02:37:22 GMT

I recommend checking out this paper: https://arxiv.org/pdf/2310.16764.pdf. From the concluding section: "Although the success of ViTs in computer vision is extremely impressive, in our view there is no strong evidence to suggest that pre-trained ViTs outperform pre-trained ConvNets when evaluated fairly." Neural networks can be unpredictable, and there's evidence that questions how important transformers' lack of inductive bias (at scale) really is.

Paper and discussion history. A mention is not an endorsement.

ConvNets Match Vision Transformers at Scale — mentioned by rocauc

Tue, 02 Apr 2024 01:38:14 GMT

Pulling out a key part of this post from a DeepMind 2023 paper[1]: “Although the success of ViTs in computer vision is extremely impressive, in our view there is no strong evidence to suggest that pre-trained ViTs outperform pre-trained ConvNets when evaluated fairly.” Another common constraint in vision vs language is the long tails are very long in the visual world. There's a number of domains where you have very little examples to learn (defects are designed to happen infrequently; rare species for identification show up, well, rarely). And pulling from the blog: "But small models ...…

Paper and discussion history. A mention is not an endorsement.

ConvNets Match Vision Transformers at Scale — mentioned by pedrosorio

Wed, 20 Mar 2024 15:59:19 GMT

ConvNets Match Vision Transformers at Scale (Oct 2023) https://arxiv.org/abs/2310.16764

Paper and discussion history. A mention is not an endorsement.

ConvNets Match Vision Transformers at Scale — HN submission

Sun, 29 Oct 2023 18:30:32 GMT

ConvNets Match Vision Transformers at Scale

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

ConvNets Match Vision Transformers at Scale — HN submission

Thu, 26 Oct 2023 22:06:26 GMT

ConvNets Match Vision Transformers at Scale

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

ConvNets Match Vision Transformers at Scale — HN submission

Thu, 26 Oct 2023 19:04:16 GMT

ConvNets Match Vision Transformers at Scale

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.