HackerPapers

arXiv:2110.09485cs.LG

Learning in High Dimension Always Amounts to Extrapolation

Randall Balestriero, Jerome Pesenti, Yann LeCun

Abstract

The notion of interpolation and extrapolation is fundamental in various fields from deep learning to function approximation. Interpolation occurs for a sample $x$ whenever this sample falls inside or on the boundary of the given dataset's convex hull. Extrapolation occurs when $x$ falls outside of that convex hull. One fundamental (mis)conception is that state-of-the-art algorithms work so well because of their ability to correctly interpolate training data. A second (mis)conception is that interpolation happens throughout tasks and datasets, in fact, many intuitions and theories rely on that assumption. We empirically and theoretically argue against those two points and demonstrate that on any high-dimensional ($>$100) dataset, interpolation almost surely never happens. Those results challenge the validity of our current interpolation/extrapolation definition as an indicator of generalization performances.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CVComputer Vision and Pattern Recognition

On Hacker News

Linked by 12 identifiable commenters in 18 comments across 16 threads, from Dec 2021 to Oct 2026. Found in 2 HN submissions from Oct 2023 to Nov 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments16 threads

  1. “Math 2.0” will need to value mathematical progress more holistically 1 comment

    • This depends on some handwavy use of the term "interpolation", not the mathematical definition. Mathematically interpolation usually means that the query point is in the convex hull of the data points, and that almost never happens in high-dimensional spaces. See: Learning in…

      bonoboTPView on HNParent context

  2. Artificial intelligence is not conscious – Ted Chiang 1 comment
  3. GPT-5.2 derives a new result in theoretical physics 1 comment

    • The tricky part is that LLMs aren't just spewing outputs from the distribution (or "near" learned manifolds), but also extrapolating / interpolating (depending on how much you care about the semantics of these terms https://arxiv.org/abs/2110.09485). There are genuine creative…

      D-MachineView on HNParent context

  4. Performance of LLMs on Advent of Code 2024 1 comment

    • I don't care much for the concept of AGI since it's ill defined. However, out of distribution generation and generalization are a better more useful metric. In these Yann Lecun has argued that interpolation is meaningless in the high dimensional spaces these "curves" are…

      redlockView on HNParent context

  5. GitHub cuts AI deals with Google, Anthropic 1 comment

    • The "statistical parrot" parrots have been demonstrably wrong for years (see e.g. LeCun et al[1]). It's just harder to ignore reality with hundreds of millions of people now using incredible new AI tools. We're approaching "don't believe your lying eyes" territory. Deniers will…

      a_wild_dandanView on HNParent context

  6. Counterintuitive Properties of High Dimensional Space (2018) 1 comment

    • If you like ML, this is also related with the results of this paper [0], where they show that learning in high dimensions amounts to extrapolation, as opposed to interpolation. Intuitively I think of this as the fact that points in the sphere are convexly independent, and most…

      youoyView on HNParent context

  7. Large language models can do jaw-dropping things. But nobody knows why 1 comment

    • I hate this as being too wooish. There first example, arithmetic, we know actually benefits from over fitting, but even then it isn't great. We now know that over Parameterization and double decent is less lossy compression. We know that feed forward networks are TC^0 and we…

      nyrikkiView on HNParent context

  8. FunSearch: Making new discoveries in mathematical sciences using LLMs 1 comment
  9. Ask HN: What is your standard for judging when AGI exists? 1 comment

    • "Supercharged Interpolation" is not a thing that exists sorry. Learning in high dimensions always results in extrapolation. https://arxiv.org/abs/2110.09485 And I would love to see the human that can generalize beyond their training distribution.

      famouswafflesView on HNParent context

  10. Pretraining data enables narrow selection capabilities in transformer models 2 comments

    • "Supercharged Interpolation" is not something that actually exists. Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 What you're asking for is not "generalization" but magic and humans would also fail.

      famouswafflesView on HNParent context

    • Learning in High Dimension Always Amounts to Extrapolation https://arxiv.org/abs/2110.09485 "Supercharged Interpolation" is not a real thing. They generalize fine when the data incentivizes that.

      famouswafflesView on HNParent context

  11. Telling GPT-4 you're scared or under pressure improves performance 2 comments

    • LLMs are not lookup tables. They aren't "matching" in that way. There's this idea of "supercharged interpolation", the idea that they just take out certain texts and switch out words or whatever, that is not true. https://arxiv.org/abs/2110.09485

      famouswafflesView on HNParent context

    • >Since it's incapable of actually summarising financial data. It's only capable of selecting combinations of pieces of its training set. Third completely off misconception from you today. This is not at all what it is doing. "Supercharged Interpolation" is false and makes no…

      famouswafflesView on HNParent context

  12. The copyright fight over authors and OpenAI's ChatGPT 1 comment

    • >That is what LLMs can do, they don't write something inspired by it they write down a copy of it with switched out words and grammar structure. That's not at all what they do. This myth of "supercharged interpolation" really needs to die. https://arxiv.org/abs/2110.09485 >It…

      famouswafflesView on HNParent context

  13. Ask HN: Have you seen anything original produced by generative AI? 1 comment

    • These models are pretty much always extrapolating [0] Whether the extrapolation is crude/low-rank or astute/high-rank is a question of memorization vs generalization. That gets into the question of whether or not the model is over-fitted or under-fitted. There are certain…

      pizzaView on HNParent context

  14. The Scaling Hypothesis (2021) 1 comment

    • I wish I could do a good job at that. Let me rob a definition from this paper "Learning in High Dimension Always Amounts to Extrapolation" https://arxiv.org/pdf/2110.09485.pdf : Definition 1. Interpolation occurs for a sample x whenever this sample belongs to the convex hull of…

      abriosiView on HNParent context

  15. Gato – A Generalist Agent 1 comment

    • There's a neat argument against these models doing interpolation: the manifold of the data is so sparse that it's infinitesimally unlikely for a good predictor to be doing interpolation between existing points on the manifold. https://arxiv.org/abs/2110.09485

      abel_View on HNParent context

  16. Training computer vision models on random noise instead of real images 1 comment

    • Depends on how you define interpolation and extrapolation. In high dimensional spaces basically everything is extrapolation including in pixel space and embedding space. > The notion of interpolation and extrapolation is fundamental in various fields from deep learning to…

      IsinlorView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.