HackerPapers

arXiv:2006.07159cs.CV

Are we done with ImageNet?

Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, Aäron van den Oord

Abstract

Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overfit to the idiosyncrasies of its labeling procedure. We therefore develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set. Using these new labels, we reassess the accuracy of recently proposed ImageNet classifiers, and find their gains to be substantially smaller than those reported on the original labels. Furthermore, we find the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end. Nevertheless, we find our annotation procedure to have largely remedied the errors in the original labels, reinforcing ImageNet as a powerful benchmark for future research in visual recognition.

arXiv comments
All five authors contributed equally. New labels at https://github.com/google-research/reassessed-imagenet
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.LGMachine Learning

On Hacker News

Linked by 3 identifiable commenters in 4 comments across 4 threads, from Sep 2020 to Sep 2021.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. Deep Learning's Diminishing Returns 1 comment

    • This number is not credible. Are we done with ImageNet?[1] points out that the best ImageNet models that currently exist surpass the accuracy of the original labels they are trained on. Expecting to reduce error by a factor 10 on unseen data points is incoherent. [1]…

      VeedracView on HNParent context

  2. Error-riddled data sets are warping our sense of how good AI is 1 comment

    • This article doesn't make any major factual errors, but it's implicating that this is some sort of surprise, and that it's a major impediment, neither or which hold up. While training on noisy datasets is an interesting problem for many practical use-cases, a few percent label…

      VeedracView on HNParent context

  3. DeepMind achieves SOTA image recognition with 8.7x faster training 1 comment

    • Not sure if I follow your JFT argument, but there's a large body of work on both (a) studying whether chasing ImageNet accuracy yields models that generalize well to out of distribution data [1, 2, 3] and (b) contextualizing progress on ImageNet (i.e., what does high accuracy on…

      loganeView on HNParent context

  4. Why neural networks struggle with the Game of Life 1 comment

    • Yes, because your validation set performance is imperfectly correlated with performance on more samples (because it's of finite size), so it will tend to overestimate within-distribution. There's also the problem that you probably want to deploy it on data which was not exactly…

      gwernView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.