Generalization in Deep Learning · HackerPapers discussions

Generalization in Deep Learning — mentioned by gradys

Wed, 30 Jun 2021 14:06:02 GMT

The AI most certainly does not lack the ability to generalize. Not as well as humans, but generalization is the key interesting result in deep learning, leading to papers like this one: https://arxiv.org/abs/1710.05468 The ability to generalize actually seems to keep increasing with the number of parameters, which is the key interesting result in the GPT-* line of work that Copilot is based on.

Paper and discussion history. A mention is not an endorsement.

Generalization in Deep Learning — mentioned by rademacher

Mon, 05 Apr 2021 12:22:27 GMT

I haven't read this paper yet so I can't speak to it's quality but it appears to be addressing the same questions in this post. Bengio is a coauthor so maybe that's a good sign . Here's the abstract. This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima, responding to an open question in the literature. We also discuss approaches to provide non-vacuous generalization guarantees for deep learning. Based on theoretical observations, we propose new open…

Paper and discussion history. A mention is not an endorsement.

Generalization in Deep Learning — mentioned by aoeusnth1

Sun, 17 Dec 2017 09:08:58 GMT

https://arxiv.org/abs/1710.05468 was an interesting paper that came out recently. It showed that large CNN models which have far greater capacity than the data they are shown (and could have memorized it) still tend to learn very good generalizable minima. See proposition 1: (i) For any model class F whose model complexity is large enough to memorize any dataset and which includes f∗ possibly at an arbitrarily sharp minimum, there exists (A, Sm) such that the generalization gap is at most epsilon, and (ii) For any dataset Sm, there exist arbitrarily unstable and arbitrarily non-robust…

Paper and discussion history. A mention is not an endorsement.

Generalization in Deep Learning — mentioned by nl

Mon, 13 Nov 2017 10:55:31 GMT

We know no more about the theory of LSTM networks than we do about ordinary NNs. Perhaps, but we do know a lot more about how (deep) NNs work than we did 40 years ago. Take for example: Generalization in Deep Learning[1] which gives a theoretical grounding of generalization. McCulloch and Pitts original 1943 paper explicitly mentions hooking a recurrent NN to external memory (they talk about a Turing machine tape). Yes, and DaVinci invented the helicopter, he just failed to get it working. [1] https://arxiv.org/pdf/1710.05468.pdf

Paper and discussion history. A mention is not an endorsement.

Generalization in Deep Learning — mentioned by nl

Tue, 31 Oct 2017 10:06:48 GMT

"boys" Not sure what this was supposed to mean? Yes, I think Fei Fei Li's datasets are much better tests than MNIST if that is what you were getting at? I don't see any of these people submitting MNIST to NIPS in 2017 None of them submitted things as entirely new and different as this, either. Having said that, I think my point holds. The completely awesome 2017 "Generalization in Deep Learning" paper[1] was co-authored by Bengio and uses MNIST - because everyone can follow it. Yann LeCun was co-author on the 2017 "Adversarially Regularized Autoencoders for Generating Discrete…

Paper and discussion history. A mention is not an endorsement.

Generalization in Deep Learning — HN submission

Tue, 17 Oct 2017 02:28:55 GMT

Generalization in Deep Learning [pdf]

126 points; 16 discussion replies on HN. Paper and source history. A submission is not an endorsement.