CvT: Introducing Convolutions to Vision Transformers · HackerPapers discussions

CvT: Introducing Convolutions to Vision Transformers — mentioned by D-Machine

Tue, 23 Dec 2025 15:09:27 GMT

Indeed, there are even multiple attempts to use both self-attention and convolutions in novel architectures, and there is evidence this works very well and may have significant advantages over pure vision transformer models [1-2]. IMO there is little reason to think transformers are (even today) the best architecture for any deep learning application. Perhaps if a mega-corp poured all their resources into some convolutional transformer architecture, you'd get something better than just the current vision transformer (ViT) models, but, since so much optimizations and work on the training of…

Paper and discussion history. A mention is not an endorsement.

CvT: Introducing Convolutions to Vision Transformers — mentioned by EnricoShippole

Thu, 14 Jul 2022 15:21:59 GMT

An open-source implementation of the CvT: Introducing Convolutions to Vision Transformers research paper in Google's JAX and Flax. JAX/Flax Repository: https://github.com/conceptofmind/CvT-flax CvT Research Paper: https://arxiv.org/abs/2103.15808 Official Github Repository: https://github.com/microsoft/CvT In collaboration with Dr. Phil 'Lucid' Wang: https://github.com/lucidrains.

Paper and discussion history. A mention is not an endorsement.

CvT: Introducing Convolutions to Vision Transformers — mentioned by tourist_on_road

Thu, 10 Mar 2022 20:41:39 GMT

Good point. The fact there is no inductive bias inherent to transformers makes it difficult to train a decent model on small datasets from scratch. However, there are recent research directions that try to address this problem [1]. Also baking in some sort of domain specific inductive bias into model architecture itself can address this problem as well [2]. [1]: Escaping the Big Data Paradigm with Compact Transformers: https://arxiv.org/abs/2104.05704 [2]: CvT: Introducing Convolutions to Vision Transformers: https://arxiv.org/abs/2103.15808

Paper and discussion history. A mention is not an endorsement.