Do ImageNet Classifiers Generalize to ImageNet? · HackerPapers discussions

Do ImageNet Classifiers Generalize to ImageNet? — mentioned by mattkrause

Mon, 06 Dec 2021 17:10:29 GMT

I disagree: some "collective overfitting" happens when everyone evaluates on the same dataset (and often, the same test set). There's a neat set of papers by Recht et al. showing results are slightly overfit to the test partitions of ImageNet and CIFAR-10: rotating examples between the train and test partitions causes systems to perform up to 10-15% worse. https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811 There's another neat bit of work involving MNIST. The original dataset (from the mid-90s) had 60,000 test examples, but the distributed versions that virtually everyone uses…

Paper and discussion history. A mention is not an endorsement.

Do ImageNet Classifiers Generalize to ImageNet? — mentioned by YeGoblynQueenne

Mon, 05 Apr 2021 19:03:27 GMT

Also see: Do ImageNet Classifiers Generalize to ImageNet? We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the…

Paper and discussion history. A mention is not an endorsement.

Do ImageNet Classifiers Generalize to ImageNet? — mentioned by logane

Sun, 14 Feb 2021 18:29:13 GMT

Not sure if I follow your JFT argument, but there's a large body of work on both (a) studying whether chasing ImageNet accuracy yields models that generalize well to out of distribution data [1, 2, 3] and (b) contextualizing progress on ImageNet (i.e., what does high accuracy on ImageNet really mean?) [4, 5, 6]. For (a), maybe surprisingly the answer is mostly yes! Better ImageNet accuracy generally corresponds to better out of distribution accuracy. For (b), it turns out that the ImageNet dataset is full of contradictions---many images have multiple ImageNet-relevant objects, and often are…

Paper and discussion history. A mention is not an endorsement.

Do ImageNet Classifiers Generalize to ImageNet? — mentioned by gwern

Thu, 24 Sep 2020 15:16:35 GMT

Yes, because your validation set performance is imperfectly correlated with performance on more samples (because it's of finite size), so it will tend to overestimate within-distribution. There's also the problem that you probably want to deploy it on data which was not exactly collected the same way. So you have issues of both internal and external validity. In practice, at least for CNN classifiers on ImageNet etc, it seems to be a pretty minor issue overall. Some links: https://arxiv.org/abs/1711.11561 https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811…

Paper and discussion history. A mention is not an endorsement.

Do ImageNet Classifiers Generalize to ImageNet? — mentioned by ajtulloch

Thu, 19 Sep 2019 23:25:43 GMT

https://arxiv.org/abs/1902.10811 is an useful counterpoint to this article's comments on ImageNet overfitting (esp e.g. §3.3, "Few Changes in the Relative Order").

Paper and discussion history. A mention is not an endorsement.

Do ImageNet Classifiers Generalize to ImageNet? — mentioned by saamm

Mon, 08 Apr 2019 21:40:40 GMT

https://arxiv.org/abs/1902.10811 There's some description in here you may find relevant.

Paper and discussion history. A mention is not an endorsement.