HackerPapers

arXiv:1806.07572cs.LG

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

Arthur Jacot, Franck Gabriel, Clément Hongler

Abstract

At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function $f_θ$ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We prove the positive-definiteness of the limiting NTK when the data is supported on the sphere and the non-linearity is non-polynomial. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function $f_θ$ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

arXiv comments
Journal reference
In Advances in neural information processing systems (pp. 8571-8580) 2018
DOI
Subjects
cs.LGMachine Learning cs.NENeural and Evolutionary Computing math.PRProbability stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 7 identifiable commenters in 9 comments across 7 threads, from Nov 2019 to Mar 2024. Found in 1 HN submission from Jan 2023 to Jan 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. Ask HN: What things are happening in ML that we can't hear over the din of LLMs? 1 comment

    • So, from the perspective I have within the subfield I work in, explainable AI (XAI), we're seeing a bunch of fascinating developments. First, as you mentioned, Rudin continues to prove that the reason for using AI/ML is that we don't understand the problem well enough; otherwise…

      babel_View on HNParent context

  2. Transformers Are Multi-State RNNs 1 comment
  3. Understanding deep learning requires rethinking generalization 2 comments

    • Why exactly is it a problem that powerful models can memorise training data with random labels, if they generalise well on non-memorisation problems? I also think that a lot of progress is being made seemingly in parallel on this "understanding NN [generalisation]" topic, and…

      medlyyyView on HNParent context

    • I believe it's this: https://arxiv.org/abs/1711.00165 And then the NTK paper: https://arxiv.org/abs/1806.07572 There is also this which seems to have been produced in parallel, closely related to the first but not exactly the same: https://arxiv.org/abs/1804.11271

      medlyyyView on HNParent context

  4. Gradient Descent Models Are Kernel Machines 1 comment

    • Quoting an expert from the last time this was posted: So at the end, it rephrased a statement from "Neural Tangent Kernel: Convergence and Generalization in Neural Networks" [https://arxiv.org/abs/1806.07572], besides in a way which is kind of miss-leading. The assertion is…

      kdisorteView on HNParent context

  5. Every Model Learned by Gradient Descent Is Approximately a Kernel Machine 1 comment

    • So at the end, it rephrased a statement from "Neural Tangent Kernel: Convergence and Generalization in Neural Networks" [https://arxiv.org/abs/1806.07572], besides in a way which is kind of miss-leading. The assertion is known by the community at least since 2018, if not even…

      FRGabrielView on HNParent context

  6. Cellular Automata as Convolutional Neural Networks 1 comment

    • What do you mean by 'the best'? Deeper architectures are popular because they quiet easy to train. They do work well in practice on many tasks (especially vision) but they have their limits. Infinite wide networks are a newly active field and has recently shown some promising…

      benibrazView on HNParent context

  7. Wide Neural Networks of Any Architecture Are Gaussian Processes 2 comments

    • For trained wide neural networks, you can have a look at "Neural Tangent Kernel: Convergence and Generalization in Neural Networks" (https://arxiv.org/abs/1806.07572) where we explain the training of very wide ANN. This sparked a numerous amount of research and in the few last…

      fgabrielView on HNParent context

    • The paper says more than just that they are GPs, I believe it shows that the NTK (Neural Tangent Kernel https://arxiv.org/abs/1806.07572) of a large class of ANN converges at initialization. This NTK allows one to gain insight on the training of NN : speed of convergence…

      fgabrielView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.