HackerPapers

arXiv:1611.03530cs.LG

Understanding deep learning requires rethinking generalization

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals

Abstract

Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family, or to the regularization techniques used during training. Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization, and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice. We interpret our experimental findings by comparison with traditional models.

arXiv comments
Published in ICLR 2017
Journal reference
DOI
Subjects
cs.LGMachine Learning

On Hacker News

Linked by 16 identifiable commenters in 18 comments across 15 threads, from Dec 2016 to Mar 2025. Found in 4 HN submissions from Feb 2017 to Sep 2020. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments15 threads

  1. Deep Learning Is Not So Mysterious or Different 2 comments

    • A decade ago the paper "Understanding deep learning requires rethinking generalization" [0] was published. The submission is a response to that paper and subsequent literature. Deep neural nets are notable for their strong generalization performance: despite being highly…

      woopwoopView on HNParent context

    • Yes, and that's the problem. What Zhang et al [2] showed convincingly in the Rethinking paper is that just focusing on the hypothesis space cannot be enough since the same hypothesis space fits real and random data so it's already too large. Therefore, these methods that focus…

      cgdlView on HNParent context

  2. Gradient descent visualization 1 comment

    • [1] Was an empirical paper that inspired much theoretical follow-up. [2] Is one such follow-up, and the references therein should point to many of the other key works in the years between. [3] Introduces the neural tangent kernel (NTK), a theoretical tool used in much of this…

      bltView on HNParent context

  3. Learning to think critically about machine learning 2 comments

    • Even if it were practically useless (which it is not, although the practical applications are less impressive than the research achievements at this point), it would be magical. Deep learning has dominated imagenet for a decade now, for example. One reason this is magical is…

      woopwoopView on HNParent context

    • With over-parameterized neural networks, the problem essentially becomes convex and even linear [1], and in many contexts provably converges to a global minimum [2], [3]. The question then becomes: why does this generalize [4], given that the classical theory of Vapnik and…

      DavidSJView on HNParent context

  4. Understanding deep learning requires rethinking generalization 1 comment

    • 2016 for v1 to be exact, link: https://arxiv.org/abs/1611.03530 and discussion: https://news.ycombinator.com/item?id=13566917

      davnnView on HNParent context

  5. Why deep learning works even though it shouldn’t 1 comment

    • I think you misunderstood the point of deep double descent. The x-axis is not number of training epochs, it is model capacity. I think you'd be interested in https://arxiv.org/abs/1611.03530. It discusses how SGD is an implicit regularizer. We also actually want high variance…

      machinelearningView on HNParent context

  6. A popular self-driving car dataset is missing labels for hundreds of pedestrians 2 comments

    • No it's not scary in the slightest. Missing or scrambled labels are routine in ML and the algorithms are able to handle it. This was a huge paper a couple years ago that demonstrates deep nets will still find the structure in the data even with totally randomized labels…

      jasonwatkinspdxView on HNParent context

    • We have a lot more than the gut feeling you're assuming: https://arxiv.org/abs/1611.03530

      jasonwatkinspdxView on HNParent context

  7. Gradient Descent Finds Global Minima of Deep Neural Networks 1 comment

    • 1. people overfit the baby datasets to zero training loss (MNIST) all the time. maybe you meant a "hard" dataset. 2. You clearly have no idea what you are talking about. This paper is trying to argue a bit about why neural networks generalize well by showing with math that a nn…

      skeptic_69View on HNParent context

  8. The Mixed Track Record of Machine Learning Algorithms 1 comment

    • Check out this classic paper on deep learning with randomized labels: https://arxiv.org/abs/1611.03530 Spoiler: the neutral net thinks it's doing a really good job!

      sdenton4View on HNParent context

  9. Teaching Computers Art History 1 comment

    • Hey I love this but watch out with this kind of stuff. And by stuff I mean training RNNs to do classification on very small (100s) sets of images. (Also you didn't mention your train/test split?). Which just means, get more data and make a smaller network by likely re-using a…

      rockmeamedeeView on HNParent context

  10. Cloud AutoML: Making AI accessible to every business 1 comment

    • I would say training a CNN from scratch or even fine-tuning one takes a lot of domain knowledge and best practices which often are not standardised yet. Besides, we don't even know why they generalise in the first place! See: https://arxiv.org/abs/1611.03530…

      eggie5View on HNParent context

  11. An Interview with an Anonymous Data Scientist (2016) 1 comment

    • Really don't think that's the best paper to say "sheds quite a bit of light on this". That paper has been somewhat controversial since it came out. I think https://arxiv.org/abs/1609.04836 is seminal in showing unsharp minima = generalization, the parent's paper is good for…

      chilleeView on HNParent context

  12. Why Deep Learning surprises me 1 comment

    • In fact recent research indicates that you can randomly relabel the training examples and the network still achieves zero training error (https://arxiv.org/abs/1611.03530). So it is not "understanding" anything intrinsic or fundamental about the letter "A". Rather, it's just…

      hyperbovineView on HNParent context

  13. Deep Learning’s Impact on Image Processing, Mathematics, and Humanity 1 comment

    • All machine learning models are capable of overfitting. Deep learning does not have a special relationship to overfitting, and in fact, there is some evidence that the structural priors in complex deep networks actually inhibit overfitting and improve generalization [1]. Citing…

      backpropagandaView on HNParent context

  14. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-Of-Experts Layer 1 comment

    • Based on our current theoretical understanding the we should expect overfitting. In practice however, a very large number of parameters does not necessarily lead to overfitting. Clearly there is a gap between out theoretical understanding and practice. Check out a recent paper…

      juresoView on HNParent context

  15. LSTM Neural Networks for Time Series Prediction 1 comment

    • Timeseries can 100% work like that. If you expect your timeseries data to be coming from a similar distribution that is what you do to train your LSTM. It's not just a magic box - you have to train it to encode useful features in the gate. Sure, you can prefer to use subsets of…

      highdView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.