Are we done with ImageNet? · HackerPapers discussions

Are we done with ImageNet? — mentioned by Veedrac

Sat, 25 Sep 2021 00:33:05 GMT

This number is not credible. Are we done with ImageNet?[1] points out that the best ImageNet models that currently exist surpass the accuracy of the original labels they are trained on. Expecting to reduce error by a factor 10 on unseen data points is incoherent. [1] https://arxiv.org/abs/2006.07159

Paper and discussion history. A mention is not an endorsement.

Are we done with ImageNet? — mentioned by Veedrac

Sun, 04 Apr 2021 12:50:04 GMT

This article doesn't make any major factual errors, but it's implicating that this is some sort of surprise, and that it's a major impediment, neither or which hold up. While training on noisy datasets is an interesting problem for many practical use-cases, a few percent label errors in these classic basic datasets (MNIST, CIFAR, ImageNet) really doesn't matter that much. The first two are basically toys to check that your system works at all at this point, and even on ImageNet the best models are now more accurate than the original labels. However, per [1], progress on current ImageNet still…

Paper and discussion history. A mention is not an endorsement.

Are we done with ImageNet? — mentioned by logane

Sun, 14 Feb 2021 18:29:13 GMT

Not sure if I follow your JFT argument, but there's a large body of work on both (a) studying whether chasing ImageNet accuracy yields models that generalize well to out of distribution data [1, 2, 3] and (b) contextualizing progress on ImageNet (i.e., what does high accuracy on ImageNet really mean?) [4, 5, 6]. For (a), maybe surprisingly the answer is mostly yes! Better ImageNet accuracy generally corresponds to better out of distribution accuracy. For (b), it turns out that the ImageNet dataset is full of contradictions---many images have multiple ImageNet-relevant objects, and often are…

Paper and discussion history. A mention is not an endorsement.

Are we done with ImageNet? — mentioned by gwern

Thu, 24 Sep 2020 15:16:35 GMT

Yes, because your validation set performance is imperfectly correlated with performance on more samples (because it's of finite size), so it will tend to overestimate within-distribution. There's also the problem that you probably want to deploy it on data which was not exactly collected the same way. So you have issues of both internal and external validity. In practice, at least for CNN classifiers on ImageNet etc, it seems to be a pretty minor issue overall. Some links: https://arxiv.org/abs/1711.11561 https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811…

Paper and discussion history. A mention is not an endorsement.