Training data-efficient image transformers & distillation through attention · HackerPapers discussions

Training data-efficient image transformers & distillation through attention — mentioned by godelski

Tue, 28 Nov 2023 18:39:13 GMT

> The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures. Has it ever? I've read a lot of "old" papers and idk if there's ever been a strong theoretical framework. I'd say it is strongest today but not in main papers and those papers are getting rejected because of lack of experiments (wtf is going on with reviewing and why are our solutions "authors have to show reviewers used LLMs"[-1] not "reviewers have to provide meaningful review"? What a joke!). Here's a more recent Universal Approximation paper[0].…

Paper and discussion history. A mention is not an endorsement.

Training data-efficient image transformers & distillation through attention — mentioned by stevenwalton

Thu, 10 Mar 2022 21:51:38 GMT

Lots of people transfer learn with transformers. ViT[0] originally did CIFAR with it. Then DeiT[1] introduced some knowledge transfer (note: their student is larger than the teacher). ViT pretrained on both ImageNet21k and JFT-300m. CCT ([1] from above) was focused on training from scratch. There's two paradigms to be aware of. ImageNet and pre-training can often be beneficial but it doesn't always help. It really depends on the problem you're trying to tackle and if there are similar features within the target dataset and the pre-trained dataset. If there is low similarity you might as well…

Paper and discussion history. A mention is not an endorsement.