HackerPapers

arXiv:2012.00152cs.LG

Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

Pedro Domingos

Abstract

Deep learning's successes are often attributed to its ability to automatically discover new representations of the data, rather than relying on handcrafted features like other learning methods. We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This greatly enhances the interpretability of deep network weights, by elucidating that they are effectively a superposition of the training examples. The network architecture incorporates knowledge of the target function into the kernel. This improved understanding should lead to better learning algorithms.

arXiv comments
12 pages, 2 figures
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.NENeural and Evolutionary Computing stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 9 identifiable commenters in 10 comments across 9 threads, from Dec 2020 to Mar 2024. Found in 6 HN submissions from Dec 2020 to Jul 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments9 threads

  1. Ask HN: What things are happening in ML that we can't hear over the din of LLMs? 1 comment

    • So, from the perspective I have within the subfield I work in, explainable AI (XAI), we're seeing a bunch of fascinating developments. First, as you mentioned, Rudin continues to prove that the reason for using AI/ML is that we don't understand the problem well enough; otherwise…

      babel_View on HNParent context

  2. Google researchers deal a major blow to the theory AI about to outsmart humans 1 comment
  3. When gradient descent is a kernel method 1 comment

    • Reminds me of "Every Model Learned by Gradient Descent Is Approximately a Kernel Machine" by Pedro Domingos: https://arxiv.org/abs/2012.00152

      JoschkabraunView on HNParent context

  4. Do Machine Learning Models Memorize or Generalize? 1 comment

    • > Statistical learning can typically be phrased in terms of k nearest neighbours We have suspected that neural nets are a kind of kNN. Here's a paper: Every Model Learned by Gradient Descent Is Approximately a Kernel Machine https://arxiv.org/abs/2012.00152

      visargaView on HNParent context

  5. Using GPT-3 to pathfind in random graphs 1 comment

    • Reminds me of Pedro Domingos's paper - Every Model Learned by Gradient Descent Is Approximately a Kernel Machine https://arxiv.org/abs/2012.00152

      visargaView on HNParent context

  6. Pathways Language Model (PaLM): Scaling to 540B parameters 2 comments

    • All GD learners are interpolators (cf. https://arxiv.org/abs/2012.00152) we also know theyre exponential in parameter count ( cf. https://www.researchgate.net/figure/Number-of-parameters-ie-... ) > If GPT is not exponential, then the m/p distinction falls apart. Yes, I think if…

      mjburgessView on HNParent context

    • > All GD learners are interpolators (cf. https://arxiv.org/abs/2012.00152) , If the paper is substantially correct I concede the point. But what I've read of reactions leads me to believe the conclusion is overstated. Regarding compression vs intelligence, I already believe that…

      FeepingCreatureView on HNParent context

  7. Deep learning is hitting a wall? 1 comment

    • Sorry, my fault: I don't mean that PAC-Learnability means that neural nets memorise their training instances. That's more my interpretation of their observed behaviour, if you like. What I meant was that PAC-Learnability doesn't assume any ability like reasoning, and really no…

      YeGoblynQueenneView on HNParent context

  8. A new link to an old model could crack the mystery of deep learning 1 comment

    • A nice, short, recent paper along these lines (not mentioned in the article) is "Every Model Learned by Gradient Descent Is Approximately a Kernel Machine": https://arxiv.org/abs/2012.00152

      homerowilsonView on HNParent context

  9. Does GPT-2 Know Your Phone Number? 1 comment

    • Thanks for the reference. Here it is: “We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for…

      antipaulView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.