Deep Double Descent: Where Bigger Models and More Data Hurt · HackerPapers discussions

Deep Double Descent: Where Bigger Models and More Data Hurt — mentioned by noosphr

Sat, 23 May 2026 12:24:23 GMT

>For example, getting good performance on a dataset with deep learning also involves a lot of guesswork. But, if your training loss is way lower than your test loss, you're in the "overfitting" regime, and you're wasting your time if you try to increase the capacity of your model. https://arxiv.org/abs/1912.02292

Paper and discussion history. A mention is not an endorsement.

Deep Double Descent: Where Bigger Models and More Data Hurt — mentioned by rndphs

Mon, 16 Feb 2026 07:09:07 GMT

https://arxiv.org/pdf/1912.02292 "We show that a variety of modern deep learning tasks exhibit a "double-descent" phenomenon where, as we increase model size, performance first gets worse and then gets better." That is the first sentence of the abstract. The first graph shown in the paper backs it up. Looking into it further, it seems that typical LLMs are in the first descent regime anyway though so my original point is not too relevant for them anyway it seems. Also it looks like the second descent region doesn't always reach a lower loss than the first, it appears to depend on other…

Paper and discussion history. A mention is not an endorsement.

Deep Double Descent: Where Bigger Models and More Data Hurt — HN submission

Sat, 27 May 2023 04:50:46 GMT

Ask HN: What are some of the best papers on AI, ML, DL and their applications?

I was reading the Deep Double Descent paper by OpenAI: https://arxiv.org/abs/1912.02292. The writing is so lucid. Even the structure of the paper is non-conventional. They have a results section before the Related Works section to give sneak peak of what the paper is about. More like telling a story. Even before starting the Introduction there is a self explanatory image spanning the entier two columns. Can you link to more papers that are written in such style and are easy to read if you have the background knowledge?

53 points; 8 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Deep Double Descent: Where Bigger Models and More Data Hurt — mentioned by sebzim4500

Wed, 26 Apr 2023 13:30:34 GMT

OpenAI paper on double descent: https://arxiv.org/pdf/1912.02292.pdf They show it happens across a variety of architectures and real world datasets

Paper and discussion history. A mention is not an endorsement.

Deep Double Descent: Where Bigger Models and More Data Hurt — mentioned by Grieverheart

Fri, 15 Oct 2021 08:06:21 GMT

The paper is a bit over my head. Are there any findings with respect to the phenomenon of Deep Double Descent [0], or the more recent grokking phenomenon [1]? [0] https://arxiv.org/abs/1912.02292 [1] https://mathai-iclr.github.io/papers/papers/MATHAI_29_paper....

Paper and discussion history. A mention is not an endorsement.

Deep Double Descent: Where Bigger Models and More Data Hurt — mentioned by cs702

Sun, 26 Jul 2020 14:55:00 GMT

> ... all DNNs face the issue of overfitting as they learn, which is when performance on one data set increases but the network's performance fails to generalize (often measured by the divergence of performance on training vs testing data sets). This ubiquitous problem in DNNs is often solved by experimenters via "noise injections" in the form of noisy or corrupted inputs. The goal of this paper is to argue that the brain faces a similar challenge of overfitting, and that nightly dreams evolved to combat the brain's overfitting during its daily learning. Actually, there's compelling evidence…

Paper and discussion history. A mention is not an endorsement.

Deep Double Descent: Where Bigger Models and More Data Hurt — HN submission

Fri, 20 Dec 2019 10:45:52 GMT

Deep Double Descent: Where Bigger Models and More Data Hurt

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Deep Double Descent: Where Bigger Models and More Data Hurt — mentioned by preetum

Sat, 07 Dec 2019 20:04:21 GMT

Hi, we define the interpolation threshold in Section 2 of the full paper (https://arxiv.org/abs/1912.02292) as the point when the "Effective Model Complexity" = # of train samples. Where the "Effective Model Complexity (EMC)" of a model + training procedure (w.r.t an input distribution) is the maximum number of samples from the distribution that the model+training can fit to ~0 train error. Our experiments are consistent with the hypothesis that the double-descent peak occurs when EMC = n; that is, when the model+training is just barely able to fit the train set.

Paper and discussion history. A mention is not an endorsement.