Learning Transferable Visual Models From Natural Language Supervision · HackerPapers discussions

Learning Transferable Visual Models From Natural Language Supervision — mentioned by kadushka

Wed, 05 Mar 2025 23:40:00 GMT

Here's a good starting point if you want to know more about this topic: https://arxiv.org/abs/2103.00020

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by montebicyclelo

Wed, 15 Jan 2025 17:54:18 GMT

The author asks: > Has the model ever seen cosine similarity? Yes - most of the time, at least for deep learning based semantic search. E.g. for semantic search of text, the majority are using, SentenceTransformers [1], models which have been trained to use cosine similarity. Or e.g. for vector representations of images, people are using models like CLIP [2], which has again been trained to use cosine similarity. (Cosine similarity being used in the training loss, so the whole model is fundamentally "tuned" for cosine similarity.) Articles like these cause confusion, e.g. I've come across…

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by ijk

Thu, 15 Aug 2024 20:43:00 GMT

Where are you getting 2 billion from? The original CLIP paper says: > We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. [1] OpenCLIP was trained on more images, but the datasets like LAION-2B are kind of low-quality in terms of labeling; I find it plausible that a better dataset could outperform it. I'm pretty sure that the stock images Adobe is drawing from have better labeling already.…

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by montebicyclelo

Tue, 12 Mar 2024 12:39:09 GMT

Cosine-similarity is a useful metric. The cases where it is useful are models that have been trained specifically to produce a meaningful cosine distance, (e.g. OpenAI's CLIP [1], Sentence Tranformers [2]) - but these are the types of models that the majority of people are using when they use cosine distances. > It's my understanding that the delta between two word embeddings, gives a direction, and the magic is from using those directions to get to new words... it's the directions and distances to nearby objects that matters most Cosine similarity is a kind of "delta" / inverse distance…

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by montebicyclelo

Tue, 12 Mar 2024 12:13:19 GMT

Hmm, typically the models where people use cosine similarity on embeddings have been deliberately trained such that the cosine similarity is meaningful. It looks like this paper is looking at examples where the models have not been deliberately trained for the cosine similarities, and hence in these situations it would indeed be unreasonable to assume cosine similarities to be a good idea.. (but that's kind of a given?) For example, here's the loss from the CLIP paper [1], which ensures cosine similarities are meaningful: # joint multimodal embedding [n, d_e] I_e = l2_normalize(np.dot(I_f…

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by RC_ITR

Tue, 18 Jul 2023 16:29:26 GMT

How is contrastive learning done with one model, exactly? I agree only one is used in inference, but two are needed for training (otherwise how do you calculate a meaningful loss function?). Notice in the original CLIP paper, there's an image encoder and a text encoder, even though only the text encoder is used during inference. [0] [0] https://arxiv.org/pdf/2103.00020.pdf

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by mwambua

Tue, 04 Oct 2022 23:28:01 GMT

From the paper on the CLIP embedder it appears that they use a form of contrastive loss that maximizes the cosine similarity between related images and prompts, but also minimizes the same between unrelated prompts & images. See section 2.3 of the CLIP paper: https://arxiv.org/pdf/2103.00020.pdf Also, the writeup on OpenAI's blog: https://openai.com/blog/clip/

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by SleekEagle

Tue, 19 Apr 2022 19:34:02 GMT

If you have a bit of background in math, I would encourage you to read the CLIP paper: https://arxiv.org/abs/2103.00020 Ultimately, the link between words and their representations comes from the CLIP training. The model generates encodings (vectors) for both an image and its corresponding caption, and then the parameters of these encoders (the functions that generate the vectors) are tuned in order to minimize the angle between the textual and visual encodings that represent the same concept. The core of your question is why minimizing the angle between like vectors is equivalent to learning…

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by SleekEagle

Tue, 19 Apr 2022 19:28:27 GMT

As far as I know, OpenAI has not made this dataset publicly available. IIRC the dataset is images scraped from instagram and their corresponding captions. Check out the CLIP paper for more details: https://arxiv.org/abs/2103.00020 Theoretically, you could build a web-scraping tool to do something like this, but even storing that data would take an absolutely insane amount of storage. I would assume OpenAI has some deal with Meta to make the creation of datasets like this easier.

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by astrange

Wed, 13 Apr 2022 02:35:49 GMT

> What is the content of the training set like, that DALL·E 2 has all this info? Web crawling. (https://arxiv.org/pdf/2103.00020.pdf 2.2) > Are there people out there spending thousands of hours just tagging or writing descriptions of training images? "Yes", Google Image Search, Wikipedia captions, Danbooru are all that in some sense. The biases should be obvious if you think about it for a minute and CLIP has even more biases since they removed all kinds of NSFW content, etc. > And does DALL·E just receive the image as-is with its description, or does it get more info (e.g. "this part of the…

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by criticaltinker

Mon, 09 Aug 2021 21:55:43 GMT

> I have discussed in this article three reasons that proves Machine Learning and Data-Driven approaches are not even relevant to NLU This is a pretty hard-line position to take, and given the authors credentials I'm inclined to believe this is somehow poorly worded and not reflective of the thesis he intended with this article. > Languages are the external artifacts that we use to encode the infinite number of thoughts that we might have. > in building larger and larger language models, Machine Learning and Data-Driven approaches are trying to chase infinity in futile attempt at trying to…

Paper and discussion history. A mention is not an endorsement.

Learning Transferable Visual Models From Natural Language Supervision — mentioned by Isinlor

Wed, 24 Mar 2021 12:09:01 GMT

Waymo is doing actual for customer self driving right now.They need to scale it up a lot, but 5-10 years is a lot of time in AI research. Deep Learning did not exist 10 years ago. Tesla is trying hard too, but it goes with a big dose of bullshiting from Musk. They are a lot more dependent on AI break troughs in DL due to their vision only philosophy. OpenAI research around scaling laws in DL suggest that naive 1000 increase in training compute over CLIP should get us to state of the art vision in zero shot settings i.e. without task specific training data. But they probably can cover 1/3 of…

Paper and discussion history. A mention is not an endorsement.