HackerPapers

arXiv:1912.02292cs.LG

Deep Double Descent: Where Bigger Models and More Data Hurt

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever

Abstract

We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better. Moreover, we show that double descent occurs not just as a function of model size, but also as a function of the number of training epochs. We unify the above phenomena by defining a new complexity measure we call the effective model complexity and conjecture a generalized double descent with respect to this measure. Furthermore, our notion of model complexity allows us to identify certain regimes where increasing (even quadrupling) the number of train samples actually hurts test performance.

arXiv comments
G.K. and Y.B. contributed equally
Journal reference
DOI
Subjects
cs.LGMachine Learning cs.CVComputer Vision and Pattern Recognition cs.NENeural and Evolutionary Computing stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 6 threads, from Dec 2019 to May 2026. Found in 2 HN submissions from Dec 2019 to May 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Making Deep Learning Go Brrrr from First Principles (2022) 1 comment

    • >For example, getting good performance on a dataset with deep learning also involves a lot of guesswork. But, if your training loss is way lower than your test loss, you're in the "overfitting" regime, and you're wasting your time if you try to increase the capacity of your…

      noosphrView on HNParent context

  2. Why I don't think AGI is imminent 1 comment

    • https://arxiv.org/pdf/1912.02292 "We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better." That is the first sentence of the abstract. The first graph shown in…

      rndphsView on HNParent context

  3. Double Descent in Human Learning 1 comment

    • OpenAI paper on double descent: https://arxiv.org/pdf/1912.02292.pdf They show it happens across a variety of architectures and real world datasets

      sebzim4500View on HNParent context

  4. Imitating Deep Learning Dynamics via Stochastic Differential Equations 1 comment

    • The paper is a bit over my head. Are there any findings with respect to the phenomenon of Deep Double Descent [0], or the more recent grokking phenomenon [1]? [0] https://arxiv.org/abs/1912.02292 [1] https://mathai-iclr.github.io/papers/papers/MATHAI_29_paper....

      GrieverheartView on HNParent context

  5. The Overfitted Brain: Dreams evolved to assist generalization 1 comment

    • > ... all DNNs face the issue of overfitting as they learn, which is when performance on one data set increases but the network's performance fails to generalize (often measured by the divergence of performance on training vs testing data sets). This ubiquitous problem in DNNs…

      cs702View on HNParent context

  6. Understanding “Deep Double Descent” 1 comment

    • Hi, we define the interpolation threshold in Section 2 of the full paper (https://arxiv.org/abs/1912.02292) as the point when the "Effective Model Complexity" = # of train samples. Where the "Effective Model Complexity (EMC)" of a model + training procedure (w.r.t an input…

      preetumView on HNParent context

Submitted to Hacker News

  • Ask HN: What are some of the best papers on AI, ML, DL and their applications?53 points8 comments

    I was reading the Deep Double Descent paper by OpenAI: https://arxiv.org/abs/1912.02292. The writing is so lucid. Even the structure of the paper is non-conventional. They have a results section before the Related Works section to give sneak peak of what the paper is about. More like telling a story. Even before starting the Introduction there is a self explanatory image spanning the entier two columns. Can you link to more papers that are written in such style and are easy to read if you have…

  • Deep Double Descent: Where Bigger Models and More Data Hurt2 points0 comments

Connected through shared HN threads, not inferred similarity or a recommendation.