HackerPapers

arXiv:2302.14045cs.CL

Language Is Not All You Need: Aligning Perception with Language Models

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei

Abstract

A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1, a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that Kosmos-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledge from language to multimodal, and from multimodal to language. In addition, we introduce a dataset of Raven IQ test, which diagnoses the nonverbal reasoning capability of MLLMs.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.CVComputer Vision and Pattern Recognition

On Hacker News

Linked by 8 identifiable commenters in 9 comments across 8 threads, from Mar 2023 to Jun 2023. Found in 2 HN submissions from Feb 2023 to Mar 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments8 threads

  1. GPT4 is 8 x 220B params = 1.7T params 1 comment

    • Probably. Check the kosmos-1 paper from Microsoft that appeared a few days before GPT4 was released: https://arxiv.org/abs/2302.14045

      benobView on HNParent context

  2. Understanding ChatGPT 1 comment

    • CLIP, which powers diffusion models, creates a joint embeddings space for text and images. There's a lot of active work on extending these multimodal embedding spaces to audio and video. Microsoft has a paper just a week or so ago showing that llm's with a joint embeddings…

      sdenton4View on HNParent context

  3. GPT-4 System Card [pdf] 1 comment

    • They probably use something similar to Kosmos-1 (https://arxiv.org/abs/2302.14045): Encode images as vectors with something like CLIP, then map them to the token space and input them between <image> </image> tags.

      benobView on HNParent context

  4. GPT-4 1 comment

    • My educated guess is that they use a MoE-style model similar to the Switch transformer[0], and combine a similar encoding as that of Kosmos-1[1] (with an “image” latch token, and a ViT-style transformer to process images). As a result, the parameter count is likely bigger, but…

      espadrineView on HNParent context

  5. GPT-4 architecture: what we can deduce from research literature 1 comment

    • It could be done in a dozen ways. One beautiful method is just using the xPos positional embedding pioneered by Microsoft and scale the context window size at runtime (even better if your attention is subquadratic - again there is a dozen of varieties to pick from), see: "A…

      kir-gadjelloView on HNParent context

  6. Show HN: BBC “In Our Time”, categorised by Dewey Decimal, heavy lifting by GPT 1 comment

    • We are months away from being able to do this with images too. All the pieces are there, and multi-modal (smallish) large language + image models are already being used in research labs; eg MS Kosmos-1[1]. Check out the visual IQ test results in the paper. Kosmos-1 is only 1.6B…

      nlView on HNParent context

  7. The Grind a Day: thousands of Apple II floppy disks archived 1 comment

    • > LLMs do not do this, and if you're training the LLMs (at cost) to do this, you're already having to do the very same searching out of materials within the corpus related to what you want. A more reasonable suggestion would be not training a LLM (which one doesn't want to do…

      gwernView on HNParent context

  8. Microsoft Kosmos-1: A Multimodal Large Language Model 2 comments

    • Paper: https://arXiv.org/abs/2302.14045 Examples: https://twitter.com/alphasignalai/status/1630651280019292161

      solaristView on HNParent context

    • Is there a better page to link to? I cannot even see "Kosmos" on this page! Edit: Ah, looks like this is the link to the paper: https://arxiv.org/abs/2302.14045 It was discussed yesterday: https://news.ycombinator.com/item?id=34965326

      tompView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.