HackerPapers

arXiv:2103.00020cs.CV

Learning Transferable Visual Models From Natural Language Supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever

Abstract

State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP.

arXiv comments
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.LGMachine Learning

On Hacker News

Linked by 9 identifiable commenters in 12 comments across 10 threads, from Mar 2021 to Mar 2025.

Comments linking this paper, per year.

Discussed in comments10 threads

  1. Ask HN: Any truly multi-modal transformer architectures? 1 comment
  2. Don't use cosine similarity carelessly 1 comment

    • The author asks: > Has the model ever seen cosine similarity? Yes - most of the time, at least for deep learning based semantic search. E.g. for semantic search of text, the majority are using, SentenceTransformers [1], models which have been trained to use cosine similarity. Or…

      montebicycleloView on HNParent context

  3. Artists score major win in copyright case against AI art generators 1 comment

    • Where are you getting 2 billion from? The original CLIP paper says: > We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million…

      ijkView on HNParent context

  4. Is Cosine-Similarity of Embeddings Really About Similarity? 2 comments

    • Hmm, typically the models where people use cosine similarity on embeddings have been deliberately trained such that the cosine similarity is meaningful. It looks like this paper is looking at examples where the models have not been deliberately trained for the cosine…

      montebicycleloView on HNParent context

    • Cosine-similarity is a useful metric. The cases where it is useful are models that have been trained specifically to produce a meaningful cosine distance, (e.g. OpenAI's CLIP [1], Sentence Tranformers [2]) - but these are the types of models that the majority of people are using…

      montebicycleloView on HNParent context

  5. fMRI-to-image with contrastive learning and diffusion priors 1 comment

    • How is contrastive learning done with one model, exactly? I agree only one is used in inference, but two are needed for training (otherwise how do you calculate a meaningful loss function?). Notice in the original CLIP paper, there's an image encoder and a text encoder, even…

      RC_ITRView on HNParent context

  6. The Illustrated Stable Diffusion 1 comment

    • From the paper on the CLIP embedder it appears that they use a form of contrastive loss that maximizes the cosine similarity between related images and prompts, but also minimizes the same between unrelated prompts & images. See section 2.3 of the CLIP paper…

      mwambuaView on HNParent context

  7. How DALL-E 2 Works 2 comments

    • As far as I know, OpenAI has not made this dataset publicly available. IIRC the dataset is images scraped from instagram and their corresponding captions. Check out the CLIP paper for more details: https://arxiv.org/abs/2103.00020 Theoretically, you could build a web-scraping…

      SleekEagleView on HNParent context

    • If you have a bit of background in math, I would encourage you to read the CLIP paper: https://arxiv.org/abs/2103.00020 Ultimately, the link between words and their representations comes from the CLIP training. The model generates encodings (vectors) for both an image and its…

      SleekEagleView on HNParent context

  8. A few things to try with DALL·E 1 comment

    • > What is the content of the training set like, that DALL·E 2 has all this info? Web crawling. (https://arxiv.org/pdf/2103.00020.pdf 2.2) > Are there people out there spending thousands of hours just tagging or writing descriptions of training images? "Yes", Google Image Search…

      astrangeView on HNParent context

  9. Machine learning won't solve natural language understanding 1 comment

    • > I have discussed in this article three reasons that proves Machine Learning and Data-Driven approaches are not even relevant to NLU This is a pretty hard-line position to take, and given the authors credentials I'm inclined to believe this is somehow poorly worded and not…

      criticaltinkerView on HNParent context

  10. The End of Tesla’s Dominance May Be Closer Than It Appears 1 comment

    • Waymo is doing actual for customer self driving right now.They need to scale it up a lot, but 5-10 years is a lot of time in AI research. Deep Learning did not exist 10 years ago. Tesla is trying hard too, but it goes with a big dose of bullshiting from Musk. They are a lot more…

      IsinlorView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.