Understanding deep learning requires rethinking generalization · HackerPapers discussions

Understanding deep learning requires rethinking generalization — mentioned by cgdl

Wed, 19 Mar 2025 04:32:42 GMT

Yes, and that's the problem. What Zhang et al [2] showed convincingly in the Rethinking paper is that just focusing on the hypothesis space cannot be enough since the same hypothesis space fits real and random data so it's already too large. Therefore, these methods that focus on the hypothesis space have to talk about a bias in practice towards a better subspace, and that already requires studying the specific optimization algorithm in order to understand why it picks certain hypothesis over others in the space. But once you are ready to do that then algorithmic stability is enough. You…

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by woopwoop

Tue, 18 Mar 2025 14:06:15 GMT

A decade ago the paper "Understanding deep learning requires rethinking generalization" [0] was published. The submission is a response to that paper and subsequent literature. Deep neural nets are notable for their strong generalization performance: despite being highly overparametrized they do not seem to overfit the training data. They still perform well on hold-out data and very often on out of distribution data "in the wild". The paper [0] noted a particularly odd feature of neural net training: one can train neural nets on standard datasets to fit random labels. There does not seem to…

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by blt

Wed, 08 May 2024 17:58:12 GMT

[1] Was an empirical paper that inspired much theoretical follow-up. [2] Is one such follow-up, and the references therein should point to many of the other key works in the years between. [3] Introduces the neural tangent kernel (NTK), a theoretical tool used in much of this work. (Not everyone agrees that reliance on NTK is the right way towards long-term theoretical progress.) [4] Is a more recent paper I haven't read yet that goes into more detail on interpolation. Its authors were well known in more "clean" parts of ML theory (e.g. bandits) and recently began studying deep learning.…

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by DavidSJ

Tue, 03 May 2022 00:16:55 GMT

With over-parameterized neural networks, the problem essentially becomes convex and even linear [1], and in many contexts provably converges to a global minimum [2], [3]. The question then becomes: why does this generalize [4], given that the classical theory of Vapnik and others [5] becomes vacuous, no longer guaranteeing lack of over-fitting? This is less well understood, although there is recent theoretical work here too. [1] Lee et al (2019). Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. https://proceedings.neurips.cc/paper/2019/hash/0d1a9651497a3...…

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by woopwoop

Mon, 02 May 2022 17:17:30 GMT

Even if it were practically useless (which it is not, although the practical applications are less impressive than the research achievements at this point), it would be magical. Deep learning has dominated imagenet for a decade now, for example. One reason this is magical is because the sota models are extremely over parametrized. There exist weights that perform perfectly on the training data but give random answers on the test data [0]. But in practice these degenerate weights are not found during sgd. What's going on there? As far as I know there is no satisfying explanation. [0]…

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by davnn

Thu, 04 Mar 2021 20:33:43 GMT

2016 for v1 to be exact, link: https://arxiv.org/abs/1611.03530 and discussion: https://news.ycombinator.com/item?id=13566917

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by machinelearning

Wed, 21 Oct 2020 22:12:54 GMT

I think you misunderstood the point of deep double descent. The x-axis is not number of training epochs, it is model capacity. I think you'd be interested in https://arxiv.org/abs/1611.03530. It discusses how SGD is an implicit regularizer. We also actually want high variance weights for symmetry breaking.

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — HN submission

Thu, 17 Sep 2020 03:34:41 GMT

Understanding deep learning requires rethinking generalization

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Understanding deep learning requires rethinking generalization — HN submission

Wed, 12 Aug 2020 17:27:13 GMT

Understanding Deep Learning Requires Re-Thinking Generalization (2017)

2 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by jasonwatkinspdx

Tue, 11 Feb 2020 21:55:05 GMT

We have a lot more than the gut feeling you're assuming: https://arxiv.org/abs/1611.03530

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by jasonwatkinspdx

Tue, 11 Feb 2020 21:50:54 GMT

No it's not scary in the slightest. Missing or scrambled labels are routine in ML and the algorithms are able to handle it. This was a huge paper a couple years ago that demonstrates deep nets will still find the structure in the data even with totally randomized labels: https://arxiv.org/abs/1611.03530 There are also various well known techniques to verify whether you've over or underfit.

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — HN submission

Sat, 15 Dec 2018 04:00:52 GMT

Understanding deep learning requires rethinking generalization

6 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by skeptic_69

Mon, 12 Nov 2018 23:06:06 GMT

1. people overfit the baby datasets to zero training loss (MNIST) all the time. maybe you meant a "hard" dataset. 2. You clearly have no idea what you are talking about. This paper is trying to argue a bit about why neural networks generalize well by showing with math that a nn with some of their conditions converges to the zero training loss. It isn't remotely meant to be practical. IT IS A THEORETICAL PAPER. And comparing it to nearest neighbors of 1 is so so so so so silly it isn't even wrong. edit. #1 is actually an entire research direction in the theory of machine learning fyi. It is…

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by sdenton4

Fri, 12 Oct 2018 15:49:51 GMT

Check out this classic paper on deep learning with randomized labels: https://arxiv.org/abs/1611.03530 Spoiler: the neutral net thinks it's doing a really good job!

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by rockmeamedee

Fri, 13 Jul 2018 15:12:16 GMT

Hey I love this but watch out with this kind of stuff. And by stuff I mean training RNNs to do classification on very small (100s) sets of images. (Also you didn't mention your train/test split?). Which just means, get more data and make a smaller network by likely re-using a trained network. I highly recommend the paper "Understanding deep learning requires rethinking generalization" (https://arxiv.org/abs/1611.03530), which shows how amazing some RNNs are at memorizing pure noise. Would love to see different applications of DL on historical visual art: GANNs making new paintings in…

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by eggie5

Wed, 17 Jan 2018 20:11:26 GMT

I would say training a CNN from scratch or even fine-tuning one takes a lot of domain knowledge and best practices which often are not standardised yet. Besides, we don't even know why they generalise in the first place! See: https://arxiv.org/abs/1611.03530, https://arxiv.org/abs/1711.11561

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by chillee

Wed, 13 Dec 2017 05:10:04 GMT

Really don't think that's the best paper to say "sheds quite a bit of light on this". That paper has been somewhat controversial since it came out. I think https://arxiv.org/abs/1609.04836 is seminal in showing unsharp minima = generalization, the parent's paper is good for showing that gradient descent over non-convex surfaces works fine, https://arxiv.org/abs/1611.03530 is landmark for kicking off this whole generalization business (mainly shows that traditional models of generalization, namely VC dimension and ideas of "capacity" don't make sense for neural nets).

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by hyperbovine

Sat, 26 Aug 2017 18:18:32 GMT

In fact recent research indicates that you can randomly relabel the training examples and the network still achieves zero training error (https://arxiv.org/abs/1611.03530). So it is not "understanding" anything intrinsic or fundamental about the letter "A". Rather, it's just storing training examples somewhere inside of its millions of parameters, which sounds a lot less impressive.

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by backpropaganda

Sun, 07 May 2017 09:38:59 GMT

All machine learning models are capable of overfitting. Deep learning does not have a special relationship to overfitting, and in fact, there is some evidence that the structural priors in complex deep networks actually inhibit overfitting and improve generalization [1]. Citing certain examples of misuse of deep learning by folks who "tweaked it till it worked" doesn't say anything about deep learning at large. The same team would do the same with a much less powerful model. Of course, being able to explain is a very valuable trait. But there's just no evidence that deep learning methods are…

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — HN submission

Sat, 04 Feb 2017 13:38:22 GMT

Understanding deep learning requires rethinking generalization

131 points; 18 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by jureso

Mon, 30 Jan 2017 05:10:10 GMT

Based on our current theoretical understanding the we should expect overfitting. In practice however, a very large number of parameters does not necessarily lead to overfitting. Clearly there is a gap between out theoretical understanding and practice. Check out a recent paper that explores exactly this questions: https://arxiv.org/abs/1611.03530

Paper and discussion history. A mention is not an endorsement.

Understanding deep learning requires rethinking generalization — mentioned by highd

Tue, 27 Dec 2016 04:12:42 GMT

Timeseries can 100% work like that. If you expect your timeseries data to be coming from a similar distribution that is what you do to train your LSTM. It's not just a magic box - you have to train it to encode useful features in the gate. Sure, you can prefer to use subsets of a single time series instead of multiple time series. The issue remains that it doesn't matter what your performance on training data is. You still need to partition your dataset into training and test data - otherwise you could just be storing a lookup table for all you know. It looks like the author has trained on…

Paper and discussion history. A mention is not an endorsement.