Better plain ViT baselines for ImageNet-1k — mentioned by EnricoShippole

Sun, 10 Jul 2022 17:02:23 GMT

An open-source implementation of the Better plain ViT baselines for ImageNet-1k research paper in Google's JAX and Flax. An update from some of the same authors of the original paper proposes simplifications to ViT that allows it to train faster and better. Among these simplifications include 2d sinusoidal positional embedding, global average pooling (no CLS token), no dropout, batch sizes of 1024 rather than 4096, and use of RandAugment and MixUp augmentations. They also show that a simple linear at the end is not significantly worse than the original MLP head. Simple ViT Research Paper…

Paper and discussion history. A mention is not an endorsement.

Better plain ViT baselines for ImageNet-1k · HackerPapers discussions

Better plain ViT baselines for ImageNet-1k — mentioned by EnricoShippole