On the Generalization Mystery in Deep Learning · HackerPapers discussions

On the Generalization Mystery in Deep Learning — mentioned by cgdl

Fri, 28 Mar 2025 16:56:13 GMT

Thank you for the great discussion. You've put your finger on the right thing I think. We can now dispense with the old VC-type thinking (i.e., that it's because the hypothesis space is not complex enough that we get generalization). Instead now the real question is this: is it the loss landscape itself, or the particular way in which the landscape is searched that leads to good generalization in deep learning. One can think of perhaps an "exhaustive" search with say God's computer of the loss landscape and pick an arbitrary point among all the points that minimize (or are close to the…

Paper and discussion history. A mention is not an endorsement.

On the Generalization Mystery in Deep Learning — mentioned by cgdl

Mon, 17 Mar 2025 17:20:51 GMT

Agreed, but PAC-Bayes or other descendants of VC theory is probably not the best explanation. The notion of algorithmic stability provides a (much) more compelling explanation. See [1] (particularly Sections 11 and 12) [1] https://arxiv.org/abs/2203.10036

Paper and discussion history. A mention is not an endorsement.

On the Generalization Mystery in Deep Learning — mentioned by cgdl

Tue, 31 Dec 2024 00:01:47 GMT

Very interesting. A related paper from a couple of years ago proposed a similar idea to understand generalization in deep learning: https://arxiv.org/abs/2203.10036

Paper and discussion history. A mention is not an endorsement.