Language Is Not All You Need: Aligning Perception with Language Models · HackerPapers discussions

Language Is Not All You Need: Aligning Perception with Language Models — mentioned by benob

Wed, 21 Jun 2023 04:56:53 GMT

Probably. Check the kosmos-1 paper from Microsoft that appeared a few days before GPT4 was released: https://arxiv.org/abs/2302.14045

Paper and discussion history. A mention is not an endorsement.

Language Is Not All You Need: Aligning Perception with Language Models — mentioned by sdenton4

Sun, 26 Mar 2023 16:35:39 GMT

CLIP, which powers diffusion models, creates a joint embeddings space for text and images. There's a lot of active work on extending these multimodal embedding spaces to audio and video. Microsoft has a paper just a week or so ago showing that llm's with a joint embeddings trained on images can do pretty amazing things, and (iirc) with better days efficiency than a text only model. These things are already here; it's just a matter of when they get out of the research labs... Which is happening fast. https://arxiv.org/abs/2302.14045

Paper and discussion history. A mention is not an endorsement.

Language Is Not All You Need: Aligning Perception with Language Models — mentioned by benob

Fri, 17 Mar 2023 13:52:12 GMT

They probably use something similar to Kosmos-1 (https://arxiv.org/abs/2302.14045): Encode images as vectors with something like CLIP, then map them to the token space and input them between <image> </image> tags.

Paper and discussion history. A mention is not an endorsement.

Language Is Not All You Need: Aligning Perception with Language Models — mentioned by espadrine

Wed, 15 Mar 2023 09:53:48 GMT

My educated guess is that they use a MoE-style model similar to the Switch transformer[0], and combine a similar encoding as that of Kosmos-1[1] (with an “image” latch token, and a ViT-style transformer to process images). As a result, the parameter count is likely bigger, but since not all of them are involved in a forward pass, it is not as meaningful. [0]: https://arxiv.org/pdf/2302.14045.pdf [1]: https://arxiv.org/pdf/2101.03961.pdf

Paper and discussion history. A mention is not an endorsement.

Language Is Not All You Need: Aligning Perception with Language Models — mentioned by kir-gadjello

Wed, 15 Mar 2023 00:29:53 GMT

It could be done in a dozen ways. One beautiful method is just using the xPos positional embedding pioneered by Microsoft and scale the context window size at runtime (even better if your attention is subquadratic - again there is a dozen of varieties to pick from), see: "A Length-Extrapolatable Transformer" https://arxiv.org/abs/2212.10554 "Language Is Not All You Need: Aligning Perception with Language Models" https://arxiv.org/abs/2302.14045 Notably, this positional embedding has been implemented by lucidrains in his x-transformers package…

Paper and discussion history. A mention is not an endorsement.

Language Is Not All You Need: Aligning Perception with Language Models — mentioned by nl

Thu, 09 Mar 2023 01:13:15 GMT

We are months away from being able to do this with images too. All the pieces are there, and multi-modal (smallish) large language + image models are already being used in research labs; eg MS Kosmos-1[1]. Check out the visual IQ test results in the paper. Kosmos-1 is only 1.6B parameters. When that or similar models scale out to 50B_ params they will be pretty amazing. [1] https://arxiv.org/abs/2302.14045

Paper and discussion history. A mention is not an endorsement.

Language Is Not All You Need: Aligning Perception with Language Models — mentioned by gwern

Wed, 08 Mar 2023 16:52:19 GMT

> LLMs do not do this, and if you're training the LLMs (at cost) to do this, you're already having to do the very same searching out of materials within the corpus related to what you want. A more reasonable suggestion would be not training a LLM (which one doesn't want to do anyway) but treating it as a retrieval+summarization task: search the corpus for mentions and similar-by-embedding documents, and summarize. LLMs are good at abstractive summarization with minimal hallucination or error. This can serve as an 'annotated bibliography', a first pass for a human writing it themselves, or the…

Paper and discussion history. A mention is not an endorsement.

Language Is Not All You Need: Aligning Perception with Language Models — HN submission

Sun, 05 Mar 2023 18:02:31 GMT

Ask HN: Have you lost enthusiasm for programming and learning after ChatGPT?

Personally, fantastic performance of LLMs and Stable Diffusion has hurt my motivation for reading documentations of cool libraries or studying another programming language book cause it seems to me all of these "classical" methods are going to be redundant soon. I mean, how far are we from interactively describing the problem in English and having an AI translate it directly to ASM instructions? Aren't LLMs, "transformers" in essence? Yeah, ChatGPT spits out hallucinations and downright wrong answers for some technical questions but it's only GPT3. Ridiculous amounts of funds is getting…

18 points; 29 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Language Is Not All You Need: Aligning Perception with Language Models — mentioned by tomp

Wed, 01 Mar 2023 10:52:13 GMT

Is there a better page to link to? I cannot even see "Kosmos" on this page! Edit: Ah, looks like this is the link to the paper: https://arxiv.org/abs/2302.14045 It was discussed yesterday: https://news.ycombinator.com/item?id=34965326

Paper and discussion history. A mention is not an endorsement.

Language Is Not All You Need: Aligning Perception with Language Models — mentioned by solarist

Wed, 01 Mar 2023 09:38:36 GMT

Paper: https://arXiv.org/abs/2302.14045 Examples: https://twitter.com/alphasignalai/status/1630651280019292161

Paper and discussion history. A mention is not an endorsement.

Language Is Not All You Need: Aligning Perception with Language Models — HN submission

Tue, 28 Feb 2023 03:36:31 GMT

Language Is Not All You Need: Aligning Perception with Language Models

87 points; 25 discussion replies on HN. Paper and source history. A submission is not an endorsement.