What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation — mentioned by mxwsn

Thu, 04 Mar 2021 20:13:19 GMT

Large model capacity enough to perfectly memorize/interpolate data may not be a bad thing. A phenomenon known as "deep double descent" says that increasing modeling capacity relative to the dataset size can reduce generalization error, even after the model achieves perfect training performance (see work by Mikhail Belkin [0] and empirical demonstrations on large deep learning tasks by researchers from Harvard/OpenAI [1]). Other work argues that memorization is critical to good performance on real-world tasks where the data distribution is often long-tailed [2]: to perform well at tasks where…

Paper and discussion history. A mention is not an endorsement.

What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation · HackerPapers discussions

What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation — mentioned by mxwsn