HackerPapers

arXiv:2010.11929cs.CV

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby

Abstract

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

arXiv comments
Fine-tuning code and pre-trained models are available at https://github.com/google-research/vision_transformer. ICLR camera-ready version with 2 small modifications: 1) Added a discussion of CLS vs GAP classifier in the appendix, 2) Fixed an error in exaFLOPs computation in Figure 5 and Table 6 (relative performance of models is basically not affected)
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.AIArtificial Intelligence cs.LGMachine Learning

On Hacker News

Linked by 15 identifiable commenters in 17 comments across 14 threads, from Jul 2021 to Sep 2026. Found in 1 HN submission from Apr 2022 to Apr 2022. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments14 threads

  1. Ask HN: (Why) Was the LLM breakthrough useful for images, audio, etc.? 1 comment

    • It's actually the other way round - the Transformer architecture was introduced for text (machine translation) in "Attention Is All You Need" (2017). Vision Transformers, which apply it to images, came three years later in 2020: https://arxiv.org/abs/2010.11929

      laruss5View on HNParent context

  2. Can you save on LLM tokens using images instead of text? 1 comment
  3. Baba Is Eval 1 comment

    • I think I remember one of the original ViT papers saying something about 2D embeddings on image patches not actually increasing performance on image recognition or segmentation, so it’s kind of interesting that it helps with text! E: I found the paper…

      ninjhaView on HNParent context

  4. Sora is here 1 comment

    • (2020) https://arxiv.org/abs/2010.11929 : an image is worth 16x16 words transformers for image recognition at scale (2021) https://arxiv.org/abs/2103.13915 : An Image is Worth 16x16 Words, What is a Video Worth? (2024) https://arxiv.org/abs/2406.07550 : An Image is Worth 32…

      GistNoesisView on HNParent context

  5. Show HN: Doctly AI – Accurate AI-Powered PDF to Markdown Parser 1 comment

    • Just tried it on this paper: https://arxiv.org/pdf/2010.11929 Pretty good results. Just wanted to know why the image url's are like this: """ ![Vision Transformer (ViT) Diagram](image-url) """ I mean it should give actual url, isn't it? Other than that, seems to work really…

      freakynitView on HNParent context

  6. The killer app of Gemini Pro 1.5 is using video as an input 3 comments

    • Back in 2020, Google was saying 16x16=256 words: https://arxiv.org/abs/2010.11929#google :)

      gwernView on HNParent context

    • This is not at all how this works. There's no separate model. Yes there's unique tokenization, if not the video as a whole then for each image. The whole video is ~1800 tokens because Gemini gets video as a series of images in context at 1 frame/s. Each image is about 258 tokens…

      famouswafflesView on HNParent context

    • >This is not at all how this works. You can literally convert the tokens returned from a video to text. What do you even think tokens are? Like seriously, before you write another word on this feel free to call the API and retrieve tokens for a video or image. Now go through the…

      llm_nerdView on HNParent context

  7. Simplifying Transformer Blocks 1 comment

    • > The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures. Has it ever? I've read a lot of "old" papers and idk if there's ever been a strong theoretical framework. I'd say it is strongest…

      godelskiView on HNParent context

  8. What is a transformer model? (2022) 2 comments

    • Just to add, the first user who replied to you is quite wrong. You can use CNNs to get features first...but it doesn't happen anymore. It's unnecessary and adds nothing. Pixels don't get fed into transformers but that's more expense than anything. Transformers need to understand…

      famouswafflesView on HNParent context

    • Vision Transformers don't use CNNs to extract anything first. (https://arxiv.org/abs/2010.11929). You could but it's not necessary and it doesn't add anything so it doesn't happen anymore. Vision transformers won't treat the image as a sequence of pixels but that's mostly…

      famouswafflesView on HNParent context

  9. The AI hype bubble is the new crypto hype bubble 1 comment

    • > AI generates text - it does nothing other than that. It's admittedly great at manipulating text, but that's where it's "general purpose transforming" stops. I will refer you to the Vision Transformer: https://arxiv.org/abs/2010.11929 I suggest you do a literature review on the…

      mxkopyView on HNParent context

  10. The Most Cited AI Papers in 2022 1 comment

    • >An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale >https://arxiv.org/abs/2010.11929 Very interesting. I didn't think of connecting these two dots. Let's see if it can be applied to other CV tasks, like object detection.

      anshumankmrView on HNParent context

  11. Imagen: An AI system that creates photorealistic images from input text 1 comment

    • I don't quite remember whether it was first used in Vit paper[1], but it's a fairly straight forward idea. You take the patches of an image like they are words in a sentence, reduce the size of the patch(num_of_pixel x num_of_pixel) with a linear projection so that we can…

      johnthewiseView on HNParent context

  12. Transformers, originally designed to handle language, are taking on vision 1 comment

    • Lots of people transfer learn with transformers. ViT[0] originally did CIFAR with it. Then DeiT[1] introduced some knowledge transfer (note: their student is larger than the teacher). ViT pretrained on both ImageNet21k and JFT-300m. CCT ([1] from above) was focused on training…

      stevenwaltonView on HNParent context

  13. Strangely Accurate AI Predictions from Blurry Medical Scans Alarms Researchers 1 comment

    • With very deep models based on Convolution (CNNs, the article is talking about AI on a computer vision problem and CNNs are the standard for these problems) you loose almost all explainability. There might be ways to partially mitigate that like adding additional "heads" to the…

      marcyb5stView on HNParent context

  14. Deep reinforcement learning is a waste of time (2019) 1 comment

    • I think you are conflating "Transformers" and "autoregressive models". Transformers are a general purpose architecture for transforming sequences into other sequences with self-attention. AR models / GANs are frameworks for generative modeling. The model architecture is almost…

      unixpickleView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.