Scaling Laws for Generative Mixed-Modal Language Models · HackerPapers discussions

Scaling Laws for Generative Mixed-Modal Language Models — mentioned by gwern

Tue, 15 Apr 2025 22:02:43 GMT

> For the first time, Liquid uncovers a scaling law that performance drop unavoidably brought by the unified training of visual and language tasks diminishes as the model size increases...No prior work has explored whether LLMs retain the power-law scaling laws observed in language tasks when extended to visual generation tasks. We prove this alignment and further show that vision can be effectively learned by LLMs as a form of language. Does this really show much that https://arxiv.org/abs/2301.03728#facebook (uncited) and other earlier work did not?

Paper and discussion history. A mention is not an endorsement.

Scaling Laws for Generative Mixed-Modal Language Models — mentioned by stevenhuang

Wed, 08 Mar 2023 02:59:14 GMT

The scaling appears linear in relationship, though on some performance metrics it is super linear. See Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682 Scaling Laws for Generative Mixed-Modal Language Models https://arxiv.org/abs/2301.03728 All this demonstrates further the "bitter lesson" http://www.incompleteideas.net/IncIdeas/BitterLesson.html that surprising ability emerges from just throwing more data and compute at the problem and is seen more fruitful an endeavor than trying to find a deeper/analytic solution to cognition (like Yann Lecun's research, who has…

Paper and discussion history. A mention is not an endorsement.

Scaling Laws for Generative Mixed-Modal Language Models — mentioned by famouswaffles

Fri, 24 Feb 2023 21:49:01 GMT

If https://arxiv.org/abs/2301.03728 And https://arxiv.org/abs/2302.00923 Is anything to go by then Large Scale + Language + Multimodality + RLHF + Toolformers

Paper and discussion history. A mention is not an endorsement.