No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance · HackerPapers discussions

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — mentioned by thesz

Mon, 09 Feb 2026 14:27:12 GMT

These diminishing returns are exponentially diminishing returns [1]. [1] https://arxiv.org/abs/2404.04125 " We consistently find that, far from exhibiting "zero-shot" generalization, multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance, following a sample inefficient log-linear scaling trend. This trend persists even when controlling for sample-level similarity between pretraining and downstream datasets, and testing on purely synthetic data distributions. Furthermore, upon benchmarking models on long-tailed data sampled based…

Paper and discussion history. A mention is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — mentioned by pton_xd

Thu, 14 Nov 2024 18:15:15 GMT

I thought I saw some paper suggesting that accuracy grows linearly with exponential data. If that's the case it's not a mystery why we'd be hitting a training wall. Not sure I got the right takeaway from that study, though. EDIT: here's the paper https://arxiv.org/abs/2404.04125

Paper and discussion history. A mention is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — mentioned by welkinSL

Thu, 25 Jul 2024 07:03:24 GMT

I wish to be proved wrong too. The current situation of "LLM" and "GPT" are very much like a hype. I cannot see how they can be used beyond problems that does NOT require accuracy and precision. IMO the problem lies on "magical thinking" that we can just say a spell and have the computer presenting the best solution ever without even knowing what we specifically want at the first space. Kind of like asking the AI for the answer to the meaning of life, the universe, and everything - "forty-two" (from <<A Hitchhiker's Guide to Galaxy>>) Personally, when I use them a lot of time I realised I am…

Paper and discussion history. A mention is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — mentioned by sweezyjeezy

Wed, 10 Jul 2024 17:07:09 GMT

Entertaining, but I think the conclusion is way off. > their vision is, at best, like that of a person with myopia seeing fine details as blurry is a crazy thing to write in an abstract. Did they try to probe that hypothesis at all? I could (well actually I can't) share some examples from my job of GPT-4v doing some pretty difficult fine-grained visual tasks that invalidate this. Personally, I rate this paper [1], which makes the argument that these huge GenAI models are pretty good at things - assuming that it has seen a LOT of that type of data during training (which is true of a great many…

Paper and discussion history. A mention is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — mentioned by sweezyjeezy

Fri, 21 Jun 2024 17:27:04 GMT

Some research to the contrary [1] - tldr is that they didn't find evidence that generative models really do zero shot well at all yet, if you show it something it literally hasn't seen before, it isn't "generally intelligent" enough to do it well. This isn't an issue for a lot of use-cases, but does seem to add some weight to the "giga-scale memorization" hypothesis. [1] https://arxiv.org/html/2404.04125v2

Paper and discussion history. A mention is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — mentioned by smartician

Fri, 14 Jun 2024 17:52:32 GMT

Research seems to suggest we need exponential training data volume increases to see meaningful performance gains: https://arxiv.org/abs/2404.04125 Personally I think we've already hit a ceiling.

Paper and discussion history. A mention is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — mentioned by thesz

Tue, 11 Jun 2024 05:13:11 GMT

> So, you're betting on no significant cost reduction of compute hardware? Seems implausible to me. This is not about compute, but about data. https://arxiv.org/abs/2404.04125 "...our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found."

Paper and discussion history. A mention is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — mentioned by intended

Fri, 07 Jun 2024 19:21:19 GMT

Ok - The most substantial claims for LLMs today come from 2 general areas - code copilots (github published, google published), and reduction in time to proficiency for new hires. That said, it may unlock additional capabilities. Additionally, language processing has a massive gap when it comes to languages that most of humanity speaks. (Gabriel Nichols, CDT paper) It appears that LLM capability growth is likely to plateau, although this is to be confirmed. https://arxiv.org/abs/2404.04125 Furthermore, having seen GenAI deployments in workplaces, they also suffer from those issues that plague…

Paper and discussion history. A mention is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — mentioned by baryphonic

Tue, 21 May 2024 00:54:37 GMT

EDIT: cite source I have yet to see any evidence of this. To the contrary, we have seen for years that advances in LLMs require orders of magnitude more parameters/power for each generation. Neural architecture search has been underwhelming. RLHF seems to have regressed models like GPT3.5+, rather than improving them. And recently, researchers concluded that multi-modal models require exponentially more data for each extra "mode" being added to the model.[0] Even in my own experimentation, I've tried to get some AGI-like behavior, and it just isn't there. I have convinced GPT4, for instance…

Paper and discussion history. A mention is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — HN submission

Thu, 09 May 2024 13:08:05 GMT

No "Zero-Shot" Without Exponential Data

187 points; 118 discussion replies on HN. Paper and source history. A submission is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — HN submission

Tue, 09 Apr 2024 14:51:36 GMT

No "Zero-Shot" Without Exponential Data

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — HN submission

Mon, 08 Apr 2024 18:56:51 GMT

No "Zero-Shot" Without Exponential Data

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance — HN submission

Mon, 08 Apr 2024 07:37:57 GMT

No "Zero-Shot" Without Exponential Data

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.