Learning in High Dimension Always Amounts to Extrapolation · HackerPapers discussions

Learning in High Dimension Always Amounts to Extrapolation — mentioned by bonoboTP

Thu, 08 Oct 2026 17:57:09 GMT

This depends on some handwavy use of the term "interpolation", not the mathematical definition. Mathematically interpolation usually means that the query point is in the convex hull of the data points, and that almost never happens in high-dimensional spaces. See: Learning in High Dimension Always Amounts to Extrapolation Randall Balestriero, Jerome Pesenti, Yann LeCun https://arxiv.org/abs/2110.09485 I guess you mean by "interpolation" that it's some kind of nonlinear combination of the training data, but that's an almost vacuous statement. Any input-output relationship has to be so by…

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by orbital-decay

Fri, 05 Jun 2026 02:27:05 GMT

Sorry, I just noticed I posted a wrong link in the comment above. Here's the proper one: https://arxiv.org/abs/2110.09485

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by D-Machine

Sat, 14 Feb 2026 07:07:00 GMT

The tricky part is that LLMs aren't just spewing outputs from the distribution (or "near" learned manifolds), but also extrapolating / interpolating (depending on how much you care about the semantics of these terms https://arxiv.org/abs/2110.09485). There are genuine creative insights that come from connecting two known semantic spaces in a way that wasn't obvious before (e.g, novel isomorphism). It is very conceivable that LLMs could make this kind of connection, but we haven't really seen a dramatic form of this yet. This kind of connection can lead to deep, non-trivial insights, but…

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by redlock

Wed, 01 Jan 2025 01:36:35 GMT

I don't care much for the concept of AGI since it's ill defined. However, out of distribution generation and generalization are a better more useful metric. In these Yann Lecun has argued that interpolation is meaningless in the high dimensional spaces these "curves" are embedded in.(https://arxiv.org/abs/2110.09485) ARC has proven they can generalize, and Alpha Go (not an llm but a deep network) has proven it can generate novel/creative solutions. We don't need AI to have a sense of self "I" for them to beat us at every human skill and activity. Infact it might be detrimental and not useful…

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by a_wild_dandan

Tue, 29 Oct 2024 18:27:26 GMT

The "statistical parrot" parrots have been demonstrably wrong for years (see e.g. LeCun et al[1]). It's just harder to ignore reality with hundreds of millions of people now using incredible new AI tools. We're approaching "don't believe your lying eyes" territory. Deniers will continue pretending that LLMs are just an NFT-level fad or bubble or whatever. The AI revolution will continue to pass them by. More's the pity. [1] https://arxiv.org/abs/2110.09485

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by youoy

Mon, 14 Oct 2024 07:08:32 GMT

If you like ML, this is also related with the results of this paper [0], where they show that learning in high dimensions amounts to extrapolation, as opposed to interpolation. Intuitively I think of this as the fact that points in the sphere are convexly independent, and most of the volume of the ball is near the boundary. [0] https://arxiv.org/abs/2110.09485

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by nyrikki

Mon, 04 Mar 2024 17:04:05 GMT

I hate this as being too wooish. There first example, arithmetic, we know actually benefits from over fitting, but even then it isn't great. We now know that over Parameterization and double decent is less lossy compression. We know that feed forward networks are TC^0 and we know that CoT etc... are limited to polynomial reductions. The fact that we can't predict exactly what is learned in a specific instance is purely a function of computational complexity theory. Attention we know is equivalent to exhaustive search. Yes interpretability is hard and math explains that. We know that the curse…

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by a_wild_dandan

Fri, 15 Dec 2023 06:53:48 GMT

"Learning in High Dimension Always Amounts to Extrapolation" [1] [1] https://arxiv.org/abs/2110.09485

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by famouswaffles

Wed, 08 Nov 2023 14:32:18 GMT

"Supercharged Interpolation" is not a thing that exists sorry. Learning in high dimensions always results in extrapolation. https://arxiv.org/abs/2110.09485 And I would love to see the human that can generalize beyond their training distribution.

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — HN submission

Tue, 07 Nov 2023 17:50:41 GMT

Learning in High Dimension Always Amounts to Extrapolation

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by famouswaffles

Tue, 07 Nov 2023 17:40:03 GMT

Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 "Supercharged Interpolation" is not a real thing. They generalize fine when the data incentivizes that.

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by famouswaffles

Tue, 07 Nov 2023 17:30:28 GMT

"Supercharged Interpolation" is not something that actually exists. Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 What you're asking for is not "generalization" but magic and humans would also fail.

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by famouswaffles

Sat, 04 Nov 2023 14:33:44 GMT

>Since it's incapable of actually summarising financial data. It's only capable of selecting combinations of pieces of its training set. Third completely off misconception from you today. This is not at all what it is doing. "Supercharged Interpolation" is false and makes no sense. It's not a lookup table either. It doesn't memorize enough of what it needs to to make your assertion possible. https://arxiv.org/abs/2110.09485

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by famouswaffles

Sat, 04 Nov 2023 11:55:32 GMT

LLMs are not lookup tables. They aren't "matching" in that way. There's this idea of "supercharged interpolation", the idea that they just take out certain texts and switch out words or whatever, that is not true. https://arxiv.org/abs/2110.09485

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by famouswaffles

Sat, 28 Oct 2023 19:49:40 GMT

>That is what LLMs can do, they don't write something inspired by it they write down a copy of it with switched out words and grammar structure. That's not at all what they do. This myth of "supercharged interpolation" really needs to die. https://arxiv.org/abs/2110.09485 >It doesn't matter if you personally aren't using them like that, the fact that they can be used like that is the problem. Of course it matters. You penalize people when they have done something wrong not if they have the capacity to.

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — HN submission

Sat, 28 Oct 2023 17:50:24 GMT

Learning in High Dimension Always Amounts to Extrapolation

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by pizza

Wed, 27 Sep 2023 01:46:00 GMT

These models are pretty much always extrapolating [0] Whether the extrapolation is crude/low-rank or astute/high-rank is a question of memorization vs generalization. That gets into the question of whether or not the model is over-fitted or under-fitted. There are certain heuristics borrowed from high dimensional statistical physics that can be used to guess how good the test performance of a model will be on a typical task without even knowing what the test data is [1]. Originality for me means finding better answers to sub-tasks, and then combining those answers together in a better way.…

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by abriosi

Tue, 31 May 2022 14:21:24 GMT

I wish I could do a good job at that. Let me rob a definition from this paper "Learning in High Dimension Always Amounts to Extrapolation" https://arxiv.org/pdf/2110.09485.pdf : Definition 1. Interpolation occurs for a sample x whenever this sample belongs to the convex hull of a set of samples X, {x1, . . . , xN }, if not, extrapolation occurs.

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by abel_

Wed, 18 May 2022 00:23:19 GMT

There's a neat argument against these models doing interpolation: the manifold of the data is so sparse that it's infinitesimally unlikely for a good predictor to be doing interpolation between existing points on the manifold. https://arxiv.org/abs/2110.09485

Paper and discussion history. A mention is not an endorsement.

Learning in High Dimension Always Amounts to Extrapolation — mentioned by Isinlor

Fri, 10 Dec 2021 12:58:34 GMT

Depends on how you define interpolation and extrapolation. In high dimensional spaces basically everything is extrapolation including in pixel space and embedding space. > The notion of interpolation and extrapolation is fundamental in various fields from deep learning to function approximation. Interpolation occurs for a sample x whenever this sample falls inside or on the boundary of the given dataset's convex hull. Extrapolation occurs when x falls outside of that convex hull. One fundamental (mis)conception is that state-of-the-art algorithms work so well because of their ability to…

Paper and discussion history. A mention is not an endorsement.