Escaping the Big Data Paradigm with Compact Transformers · HackerPapers discussions

Escaping the Big Data Paradigm with Compact Transformers — mentioned by godelski

Thu, 24 Jul 2025 00:54:36 GMT

Personally, I didn't find too much value in this paper. I think it is good as a product demonstration, I just don't know if it added a ton of value into the research space (but maybe it did because people have been making the same mistake for awhile?). I actually think the linked page makes it very easy to understand my main critique. The main problem here is that downscaling is a destructive process. It destroys information. Zoom in on that sign, can you read it?[0] No! But can you in the high res?[1] Of course! We can of course train the model on those signs alone and then get it to…

Paper and discussion history. A mention is not an endorsement.

Escaping the Big Data Paradigm with Compact Transformers — mentioned by stevenwalton

Sat, 02 Dec 2023 04:24:38 GMT

Logged into my personal account for this one! I'm a lead author on a paper that explored exactly. It does enable faster training and smaller model sizes. For reference, you can get 80% accuracy on CIFAR-10 in ~30 minutes of CPU (not using crazy optimizations). There are open questions about scaling but at the time we did not have access to big compute (really still don't) and our goals were focused on addressing the original ViT's claims of data constraints and necessities of pretraining for smaller datasets (spoiler, augmentation + overlapping patches plays a huge role). Basically we wanted…

Paper and discussion history. A mention is not an endorsement.

Escaping the Big Data Paradigm with Compact Transformers — mentioned by tourist_on_road

Thu, 10 Mar 2022 20:41:39 GMT

Good point. The fact there is no inductive bias inherent to transformers makes it difficult to train a decent model on small datasets from scratch. However, there are recent research directions that try to address this problem [1]. Also baking in some sort of domain specific inductive bias into model architecture itself can address this problem as well [2]. [1]: Escaping the Big Data Paradigm with Compact Transformers: https://arxiv.org/abs/2104.05704 [2]: CvT: Introducing Convolutions to Vision Transformers: https://arxiv.org/abs/2103.15808

Paper and discussion history. A mention is not an endorsement.