Reconciling modern machine learning practice and the bias-variance trade-off · HackerPapers discussions

Reconciling modern machine learning practice and the bias-variance trade-off — HN submission

Sat, 23 Apr 2022 00:48:17 GMT

Reconciling modern machine learning practice and the bias-variance trade-off

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Reconciling modern machine learning practice and the bias-variance trade-off — mentioned by blackbear_

Mon, 05 Apr 2021 13:08:24 GMT

And yet that is not what is observed in practice. See figures 1 and 2 of that paper [1]. What I am complaining about is that the authors are confusing "what can be represented" by a neural network with "what can be easily learned" via SGD. I argued that the peak in test loss in those figures is observed becuse SGD struggles to find a solution that generalizes well, not because those networks are intrinsically less powerful (as the paper seems to imply). [1] https://arxiv.org/pdf/1812.11118.pdf

Paper and discussion history. A mention is not an endorsement.

Reconciling modern machine learning practice and the bias-variance trade-off — mentioned by mxwsn

Thu, 04 Mar 2021 20:13:19 GMT

Large model capacity enough to perfectly memorize/interpolate data may not be a bad thing. A phenomenon known as "deep double descent" says that increasing modeling capacity relative to the dataset size can reduce generalization error, even after the model achieves perfect training performance (see work by Mikhail Belkin [0] and empirical demonstrations on large deep learning tasks by researchers from Harvard/OpenAI [1]). Other work argues that memorization is critical to good performance on real-world tasks where the data distribution is often long-tailed [2]: to perform well at tasks where…

Paper and discussion history. A mention is not an endorsement.

Reconciling modern machine learning practice and the bias-variance trade-off — mentioned by cs702

Tue, 20 Oct 2020 21:11:34 GMT

TL;DR: For high-dimensional models (say, with millions to billions of parameters), there's always a good set parameters nearby, and when we start descending towards it, we are highly unlikely to get stuck, because almost always there's at least one path down along at least one among of all those dimensions -- i.e., there are no local optima. Once we've stumbled upon a good set of parameters, as measured by validation, we can stop. These intuitions are consistent with my experience... but I think there's more to deep learning. For instance, these intuitions fail to explain "weird" phenomena…

Paper and discussion history. A mention is not an endorsement.

Reconciling modern machine learning practice and the bias-variance trade-off — HN submission

Tue, 04 Feb 2020 15:32:33 GMT

Reconciling modern machine learning practice and the bias-variance trade-off

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Reconciling modern machine learning practice and the bias-variance trade-off — HN submission

Tue, 27 Aug 2019 01:23:32 GMT

Reconciling modern machine learning and the bias-variance trade-off

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.