An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale · HackerPapers discussions

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by laruss5

Mon, 07 Sep 2026 09:18:27 GMT

It's actually the other way round - the Transformer architecture was introduced for text (machine translation) in "Attention Is All You Need" (2017). Vision Transformers, which apply it to images, came three years later in 2020: https://arxiv.org/abs/2010.11929

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by heltale

Sat, 08 Nov 2025 05:13:35 GMT

I suppose it’s only worth 256 words at a time right now. ;) https://arxiv.org/abs/2010.11929

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by ninjha

Sat, 05 Jul 2025 12:14:24 GMT

I think I remember one of the original ViT papers saying something about 2D embeddings on image patches not actually increasing performance on image recognition or segmentation, so it’s kind of interesting that it helps with text! E: I found the paper: https://arxiv.org/pdf/2010.11929 > We use standard learnable 1D position embeddings, since we have not observed significant performance gains from using more advanced 2D-aware position embeddings (Appendix D.4). Although it looks like that was just ImageNet so maybe this isn't that surprising.

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by GistNoesis

Mon, 09 Dec 2024 18:49:20 GMT

(2020) https://arxiv.org/abs/2010.11929 : an image is worth 16x16 words transformers for image recognition at scale (2021) https://arxiv.org/abs/2103.13915 : An Image is Worth 16x16 Words, What is a Video Worth? (2024) https://arxiv.org/abs/2406.07550 : An Image is Worth 32 Tokens for Reconstruction and Generation

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by freakynit

Sat, 26 Oct 2024 16:45:45 GMT

Just tried it on this paper: https://arxiv.org/pdf/2010.11929 Pretty good results. Just wanted to know why the image url's are like this: """ ![Vision Transformer (ViT) Diagram](image-url) """ I mean it should give actual url, isn't it? Other than that, seems to work really well. Congrats..

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by llm_nerd

Thu, 22 Feb 2024 13:51:05 GMT

>This is not at all how this works. You can literally convert the tokens returned from a video to text. What do you even think tokens are? Like seriously, before you write another word on this feel free to call the API and retrieve tokens for a video or image. Now go through the magical process of converting those tokens back to their text form. It isn't some magical hyper-dimensional, inside-out spatial encoding that yields impossible compression. This process is obvious and logical if actually thought through. >Each image is about 258 tokens Because Google set that as the "budget" and…

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by famouswaffles

Thu, 22 Feb 2024 13:43:24 GMT

This is not at all how this works. There's no separate model. Yes there's unique tokenization, if not the video as a whole then for each image. The whole video is ~1800 tokens because Gemini gets video as a series of images in context at 1 frame/s. Each image is about 258 tokens because a token in image transformer terms is literally a patch of the image. https://arxiv.org/abs/2010.11929

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by gwern

Wed, 21 Feb 2024 22:09:02 GMT

Back in 2020, Google was saying 16x16=256 words: https://arxiv.org/abs/2010.11929#google :)

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by godelski

Tue, 28 Nov 2023 18:39:13 GMT

> The field of ML research is moving so fast that people don’t even take time anymore to explain the design decisions behind their architectures. Has it ever? I've read a lot of "old" papers and idk if there's ever been a strong theoretical framework. I'd say it is strongest today but not in main papers and those papers are getting rejected because of lack of experiments (wtf is going on with reviewing and why are our solutions "authors have to show reviewers used LLMs"[-1] not "reviewers have to provide meaningful review"? What a joke!). Here's a more recent Universal Approximation paper[0].…

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by famouswaffles

Fri, 23 Jun 2023 20:33:30 GMT

Vision Transformers don't use CNNs to extract anything first. (https://arxiv.org/abs/2010.11929). You could but it's not necessary and it doesn't add anything so it doesn't happen anymore. Vision transformers won't treat the image as a sequence of pixels but that's mostly because doing that gets very expensive very fast. The image is split into patches and the patches have positional embeddings.

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by famouswaffles

Fri, 23 Jun 2023 20:31:17 GMT

Just to add, the first user who replied to you is quite wrong. You can use CNNs to get features first...but it doesn't happen anymore. It's unnecessary and adds nothing. Pixels don't get fed into transformers but that's more expense than anything. Transformers need to understand how each piece relates to every other piece. That gets very costly very fast when the "pieces" are pixels. Images are split into patches instead with positional embeddings. as for how it learns the representations anyway. Well it's not like there's any specific intuition to it. after all, the original authors didn't…

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by mxkopy

Sun, 11 Jun 2023 23:29:06 GMT

> AI generates text - it does nothing other than that. It's admittedly great at manipulating text, but that's where it's "general purpose transforming" stops. I will refer you to the Vision Transformer: https://arxiv.org/abs/2010.11929 I suggest you do a literature review on the applications of ANNs, CNNs and attention mechanisms. They are quite numerous but I will concede that they generally 'just' do compression. AI encodes human concepts like emotions and spatial reasoning into vectors. Which, their analogous-ness to the real world, is more of a philosophical question than a practical one…

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by anshumankmr

Mon, 06 Mar 2023 03:05:59 GMT

>An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale >https://arxiv.org/abs/2010.11929 Very interesting. I didn't think of connecting these two dots. Let's see if it can be applied to other CV tasks, like object detection.

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by johnthewise

Thu, 25 Aug 2022 09:51:15 GMT

I don't quite remember whether it was first used in Vit paper[1], but it's a fairly straight forward idea. You take the patches of an image like they are words in a sentence, reduce the size of the patch(num_of_pixel x num_of_pixel) with a linear projection so that we can actually process it and get rid of sparse pixel information, add in positional encodings to put in location information of the patch and treat them as how you treated words in language models from that point on with transformers. Essentially, words are human constructed but information dense representation of language but…

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — HN submission

Fri, 01 Apr 2022 06:52:13 GMT

An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale

8 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by stevenwalton

Thu, 10 Mar 2022 21:51:38 GMT

Lots of people transfer learn with transformers. ViT[0] originally did CIFAR with it. Then DeiT[1] introduced some knowledge transfer (note: their student is larger than the teacher). ViT pretrained on both ImageNet21k and JFT-300m. CCT ([1] from above) was focused on training from scratch. There's two paradigms to be aware of. ImageNet and pre-training can often be beneficial but it doesn't always help. It really depends on the problem you're trying to tackle and if there are similar features within the target dataset and the pre-trained dataset. If there is low similarity you might as well…

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by marcyb5st

Fri, 27 Aug 2021 06:33:12 GMT

With very deep models based on Convolution (CNNs, the article is talking about AI on a computer vision problem and CNNs are the standard for these problems) you loose almost all explainability. There might be ways to partially mitigate that like adding additional "heads" to the model to output some debugging-like data, or run the image in patches and see where the model is more confident about a particular metric of interest. Both these options are not optimal as you still don't see the inner workings of the model in an interpretable way. Transformers models [1], on the other hand, are the…

Paper and discussion history. A mention is not an endorsement.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — mentioned by unixpickle

Sat, 10 Jul 2021 21:23:24 GMT

I think you are conflating "Transformers" and "autoregressive models". Transformers are a general purpose architecture for transforming sequences into other sequences with self-attention. AR models / GANs are frameworks for generative modeling. The model architecture is almost entirely orthogonal to the generative framework. You can use transformers as part of GANs [1], and you can even use them as discriminative models for images [2]. [1] https://arxiv.org/abs/2102.07074 [2] https://arxiv.org/abs/2010.11929

Paper and discussion history. A mention is not an endorsement.