Do CIFAR-10 Classifiers Generalize to CIFAR-10? · HackerPapers discussions

Do CIFAR-10 Classifiers Generalize to CIFAR-10? — mentioned by iflp

Wed, 18 Jan 2023 13:55:20 GMT

No. I was referring to the "standard concentration bound" in that paper, which applies when you have separate validation and test sets. I think the argument can usually be improved by applying small-variance inequalities such as Bernstein's, to excess risk-like quantities such as l(f_hat(x), y) - l(f_ref(x), y), to show that accuracy difference / relative rank enjoys better guarantees. For ImageNet we can use the 01 loss and set f_{ref} to a SoTA classifier which, while having its loss bounded away from 0, is "mostly similar" to most f_hat's, and thus leads to a small excess risk. The CIFAR…

Paper and discussion history. A mention is not an endorsement.

Do CIFAR-10 Classifiers Generalize to CIFAR-10? — mentioned by mattkrause

Mon, 06 Dec 2021 17:10:29 GMT

I disagree: some "collective overfitting" happens when everyone evaluates on the same dataset (and often, the same test set). There's a neat set of papers by Recht et al. showing results are slightly overfit to the test partitions of ImageNet and CIFAR-10: rotating examples between the train and test partitions causes systems to perform up to 10-15% worse. https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811 There's another neat bit of work involving MNIST. The original dataset (from the mid-90s) had 60,000 test examples, but the distributed versions that virtually everyone uses…

Paper and discussion history. A mention is not an endorsement.

Do CIFAR-10 Classifiers Generalize to CIFAR-10? — mentioned by gwern

Thu, 24 Sep 2020 15:16:35 GMT

Yes, because your validation set performance is imperfectly correlated with performance on more samples (because it's of finite size), so it will tend to overestimate within-distribution. There's also the problem that you probably want to deploy it on data which was not exactly collected the same way. So you have issues of both internal and external validity. In practice, at least for CNN classifiers on ImageNet etc, it seems to be a pretty minor issue overall. Some links: https://arxiv.org/abs/1711.11561 https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811…

Paper and discussion history. A mention is not an endorsement.

Do CIFAR-10 Classifiers Generalize to CIFAR-10? — mentioned by seesawtron

Fri, 05 Jun 2020 07:51:10 GMT

A similar study in the past (2018) "Do CIFAR-10 Classifiers Generalize to CIFAR-10?": https://arxiv.org/abs/1806.00451

Paper and discussion history. A mention is not an endorsement.

Do CIFAR-10 Classifiers Generalize to CIFAR-10? — HN submission

Sun, 27 Jan 2019 14:25:28 GMT

Do CIFAR-10 Classifiers Generalize to CIFAR-10?

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Do CIFAR-10 Classifiers Generalize to CIFAR-10? — mentioned by asparagui

Sat, 01 Dec 2018 15:20:35 GMT

Yes, over-fitting is definitely something that happens. Here's a paper that demonstrates the problem by testing CIFAR classifiers on a new version of the input data: https://arxiv.org/abs/1806.00451 The flip side is that by having a common problem to tackle it's much easier to compare and contrast different results to figure out what really works. Many approaches that improve results on CIFAR don't scale to ImageNet, and many of the ImageNet papers don't scale to the even larger datasets that people are using now.

Paper and discussion history. A mention is not an endorsement.

Do CIFAR-10 Classifiers Generalize to CIFAR-10? — mentioned by Dzugaru

Tue, 14 Aug 2018 14:40:57 GMT

Many years of "human optimization" on the same old datasets must lead to severe overfitting, so these results should be taken with a grain of salt. Related research: https://arxiv.org/abs/1806.00451

Paper and discussion history. A mention is not an endorsement.

Do CIFAR-10 Classifiers Generalize to CIFAR-10? — mentioned by dkislyuk

Mon, 11 Jun 2018 00:33:12 GMT

Treating training datasets as dynamic components of ML systems, along with the corresponding tooling and infrastructure, is one of the most under-appreciated points in the field today. Part of the cause, I think, is due to stigma coming from the academic side that dataset collection is a low-level problem not worthy of serious algorithmic investment (good luck submitting a CVPR paper which improves on labeling speed or taxonomy management in your dataset warehouse). Datasets are considered a given, which also has the interesting side effect of massive hyperparameter and architecture…

Paper and discussion history. A mention is not an endorsement.

Do CIFAR-10 Classifiers Generalize to CIFAR-10? — mentioned by Eridrus

Sun, 10 Jun 2018 18:47:53 GMT

This paper is definitely quite interesting, but before this thread turns into a bunch of NN hate, here is another take: "Do CIFAR-10 Classifiers Generalize to CIFAR-10?" - https://arxiv.org/abs/1806.00451 They use the same procedure used to construct CIFAR-10 to construct a new test set and test a bunch of state of the art results on the new test set. They see a generalization gap, but the relative order of SoTA results remains (roughly) the same. So, yes, test set validation is an overestimate of real-world performance of these systems, but progress on test sets is indicative of progress in…

Paper and discussion history. A mention is not an endorsement.

Do CIFAR-10 Classifiers Generalize to CIFAR-10? — HN submission

Mon, 04 Jun 2018 19:01:11 GMT

Do CIFAR-10 Classifiers Generalize to CIFAR-10? [pdf]

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.