Linearly Mapping from Image to Text Space · HackerPapers discussions

Linearly Mapping from Image to Text Space — mentioned by famouswaffles

Thu, 07 Dec 2023 10:27:53 GMT

>the sounds for man, and the word man can be mapped. The concept of man can be mapped from English to Chinese. Given enough of these clues, and a large enough data set it's not shocking that models get decent at inference between languages. 1. Many such concepts cannot be directly mapped between different languages especially with distant language pairs. 2. Language Models and Image models even if both are only trained on their respective modality learn structural representations so similar, you can connect them with a simple linear layer. That's it. Entirely different modalities…

Paper and discussion history. A mention is not an endorsement.

Linearly Mapping from Image to Text Space — mentioned by JoshuaDavid

Wed, 14 Jun 2023 01:48:01 GMT

Slightly off-topic, but I am just starting to learn about the field, and saw > GPT4 still projects information into some highly non-linear latent space and samples from that Do you have more information/know where I can find things to read about the "highly non-linear" part of that? I've been reading some stuff[1][2][3] about smaller models, and my impression has been that the latent space is shockingly linear for those ones. But counterexamples would be informative, especially if it's something that only happens with larger models. ----- [1] https://arxiv.org/abs/2209.15162 [2]…

Paper and discussion history. A mention is not an endorsement.

Linearly Mapping from Image to Text Space — mentioned by famouswaffles

Wed, 19 Apr 2023 15:47:34 GMT

There's something a bit more mindblowing than that. Language models and vision models learn representations so similar that you can connect them with just a linear projection between image embedding and text embedding space(no training of the image encoder or llm required). https://arxiv.org/abs/2209.15162 https://llava-vl.github.io/ LLMs are already being grounded.

Paper and discussion history. A mention is not an endorsement.

Linearly Mapping from Image to Text Space — mentioned by famouswaffles

Wed, 19 Apr 2023 01:03:56 GMT

as well as this - https://llava-vl.github.io/, Just found this paper that demonstrated this a few months ago (that somehow language and vision models learn representations similar enough that linear projection is enough) https://arxiv.org/abs/2209.15162

Paper and discussion history. A mention is not an endorsement.

Linearly Mapping from Image to Text Space — mentioned by famouswaffles

Tue, 18 Apr 2023 23:15:10 GMT

Wow. So far this seems better than mini-GPT4. The most impressive thing is how little training data this took. and the fact that you only need CLIP and a linear projection is wild. at least with mini-GPT-4, you could say that BLIP-2 already translates RGB into text token space and maybe a simple linear projection should work. But that you can project the image encoding directly...wild. EDIT - Just found this paper that demonstrated this a few months ago (that somehow language and vision models learn representations similar enough that linear projection is enough)…

Paper and discussion history. A mention is not an endorsement.

Linearly Mapping from Image to Text Space — mentioned by visarga

Fri, 07 Oct 2022 21:58:26 GMT

The hypothesis that you can't learn some things from text - you need real life experience, is intuitive and I used to think it's true. But there are interesting results from just a few days ago saying that text by itself is also enough: > We test a stronger hypothesis: that the conceptual representations learned by text only models are functionally equivalent (up to a linear transformation) to those learned by models trained on vision tasks. Specifically, we show that the image representations from vision models can be transferred as continuous prompts to frozen LMs by training only a single…

Paper and discussion history. A mention is not an endorsement.