HackerPapers

arXiv:1902.10811cs.CV

Do ImageNet Classifiers Generalize to ImageNet?

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar

Abstract

We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly "harder" images than those found in the original test sets.

arXiv comments
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 6 identifiable commenters in 6 comments across 6 threads, from Apr 2019 to Dec 2021.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. A cartel of influential datasets are dominating machine learning research 1 comment

    • I disagree: some "collective overfitting" happens when everyone evaluates on the same dataset (and often, the same test set). There's a neat set of papers by Recht et al. showing results are slightly overfit to the test partitions of ImageNet and CIFAR-10: rotating examples…

      mattkrauseView on HNParent context

  2. Are deep neural networks dramatically overfitted? (2019) 1 comment

    • Also see: Do ImageNet Classifiers Generalize to ImageNet? We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely…

      YeGoblynQueenneView on HNParent context

  3. DeepMind achieves SOTA image recognition with 8.7x faster training 1 comment

    • Not sure if I follow your JFT argument, but there's a large body of work on both (a) studying whether chasing ImageNet accuracy yields models that generalize well to out of distribution data [1, 2, 3] and (b) contextualizing progress on ImageNet (i.e., what does high accuracy on…

      loganeView on HNParent context

  4. Why neural networks struggle with the Game of Life 1 comment

    • Yes, because your validation set performance is imperfectly correlated with performance on more samples (because it's of finite size), so it will tend to overestimate within-distribution. There's also the problem that you probably want to deploy it on data which was not exactly…

      gwernView on HNParent context

  5. AI competitions don’t produce useful models 1 comment

    • https://arxiv.org/abs/1902.10811 is an useful counterpoint to this article's comments on ImageNet overfitting (esp e.g. §3.3, "Few Changes in the Relative Order").

      ajtullochView on HNParent context

  6. Ask HN: Dataset Cleaning Tips 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.