Neural Tangent Kernel: Convergence and Generalization in Neural Networks · HackerPapers discussions

Neural Tangent Kernel: Convergence and Generalization in Neural Networks — mentioned by babel_

Thu, 28 Mar 2024 13:59:52 GMT

So, from the perspective I have within the subfield I work in, explainable AI (XAI), we're seeing a bunch of fascinating developments. First, as you mentioned, Rudin continues to prove that the reason for using AI/ML is that we don't understand the problem well enough; otherwise we wouldn't even think to use it! So, pushing our focus to better understand the problem, and then levy ML concepts and techniques (including "classical AI" and statistical learning), we're able to make something that not only outperforms some state-of-the-art in most metrics, but often even is much less resource…

Paper and discussion history. A mention is not an endorsement.

Neural Tangent Kernel: Convergence and Generalization in Neural Networks — mentioned by blackbear_

Tue, 16 Jan 2024 21:34:47 GMT

I hope you're satisfied with Gaussian Processes: https://arxiv.org/abs/1806.07572

Paper and discussion history. A mention is not an endorsement.

Neural Tangent Kernel: Convergence and Generalization in Neural Networks — HN submission

Thu, 26 Jan 2023 13:37:31 GMT

Neural Tangent Kernel: Convergence and Generalization in Neural Networks

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Neural Tangent Kernel: Convergence and Generalization in Neural Networks — mentioned by medlyyy

Fri, 05 Mar 2021 20:34:58 GMT

I believe it's this: https://arxiv.org/abs/1711.00165 And then the NTK paper: https://arxiv.org/abs/1806.07572 There is also this which seems to have been produced in parallel, closely related to the first but not exactly the same: https://arxiv.org/abs/1804.11271

Paper and discussion history. A mention is not an endorsement.

Neural Tangent Kernel: Convergence and Generalization in Neural Networks — mentioned by medlyyy

Fri, 05 Mar 2021 11:55:54 GMT

Why exactly is it a problem that powerful models can memorise training data with random labels, if they generalise well on non-memorisation problems? I also think that a lot of progress is being made seemingly in parallel on this "understanding NN [generalisation]" topic, and the different threads don't necessarily seem to be aware of each other. This article seems to have failed to mention most of this recent progress. Possibly this is an artifact of the time it takes to produce and publish a completed article, but it presents a picture that we are more clueless than we actually are. The…

Paper and discussion history. A mention is not an endorsement.

Neural Tangent Kernel: Convergence and Generalization in Neural Networks — mentioned by kdisorte

Mon, 08 Feb 2021 21:58:09 GMT

Quoting an expert from the last time this was posted: So at the end, it rephrased a statement from "Neural Tangent Kernel: Convergence and Generalization in Neural Networks" [https://arxiv.org/abs/1806.07572], besides in a way which is kind of miss-leading. The assertion is known by the community at least since 2018, if not even well before. I find this article and the buzz around a little awkward.

Paper and discussion history. A mention is not an endorsement.

Neural Tangent Kernel: Convergence and Generalization in Neural Networks — mentioned by FRGabriel

Sun, 06 Dec 2020 20:10:46 GMT

So at the end, it rephrased a statement from "Neural Tangent Kernel: Convergence and Generalization in Neural Networks" [https://arxiv.org/abs/1806.07572], besides in a way which is kind of miss-leading. The assertion is known by the community at least since 2018, if not even well before. I find this article and the buzz around a little awkward.

Paper and discussion history. A mention is not an endorsement.

Neural Tangent Kernel: Convergence and Generalization in Neural Networks — mentioned by benibraz

Thu, 13 Aug 2020 09:01:47 GMT

What do you mean by 'the best'? Deeper architectures are popular because they quiet easy to train. They do work well in practice on many tasks (especially vision) but they have their limits. Infinite wide networks are a newly active field and has recently shown some promising results, theoretically [1, 2] and empirically [3]. [1] https://arxiv.org/abs/2001.06931 [3] https://arxiv.org/abs/1806.07572 [2] https://ai.googleblog.com/2020/03/fast-and-easy-infinitely-w...

Paper and discussion history. A mention is not an endorsement.

Neural Tangent Kernel: Convergence and Generalization in Neural Networks — mentioned by fgabriel

Thu, 28 Nov 2019 06:38:08 GMT

The paper says more than just that they are GPs, I believe it shows that the NTK (Neural Tangent Kernel https://arxiv.org/abs/1806.07572) of a large class of ANN converges at initialization. This NTK allows one to gain insight on the training of NN : speed of convergence, artifacts which can appear (checkerboard patterns can be explained with NTK), generalisation of the NN... Would be nice to get the same result during training for all these architectures and I believe it will be the next paper of G. Yang and I eager to read it.

Paper and discussion history. A mention is not an endorsement.

Neural Tangent Kernel: Convergence and Generalization in Neural Networks — mentioned by fgabriel

Thu, 28 Nov 2019 06:30:28 GMT

For trained wide neural networks, you can have a look at "Neural Tangent Kernel: Convergence and Generalization in Neural Networks" (https://arxiv.org/abs/1806.07572) where we explain the training of very wide ANN. This sparked a numerous amount of research and in the few last results about training wide ANN you can have a look at: https://arxiv.org/abs/1909.08156 and https://arxiv.org/abs/1904.11955

Paper and discussion history. A mention is not an endorsement.