HackerPapers

arXiv:1806.00451cs.LG

Do CIFAR-10 Classifiers Generalize to CIFAR-10?

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar

Abstract

Machine learning is currently dominated by largely experimental work focused on improvements in a few key tasks. However, the impressive accuracy numbers of the best performing models are questionable because the same test sets have been used to select these models for multiple years now. To understand the danger of overfitting, we measure the accuracy of CIFAR-10 classifiers by creating a new test set of truly unseen images. Although we ensure that the new test set is as close to the original data distribution as possible, we find a large drop in accuracy (4% to 10%) for a broad range of deep learning models. Yet more recent models with higher original accuracy show a smaller drop and better overall performance, indicating that this drop is likely not due to overfitting based on adaptivity. Instead, we view our results as evidence that current accuracy numbers are brittle and susceptible to even minute natural variations in the data distribution.

arXiv comments
Journal reference
DOI
Subjects
cs.LGMachine Learning stat.MLMachine Learning (Statistics)

On Hacker News

Linked by 8 identifiable commenters in 8 comments across 8 threads, from Jun 2018 to Jan 2023. Found in 2 HN submissions from Jun 2018 to Jan 2019. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments8 threads

  1. A stack of feed-forward layers does surprisingly well on ImageNet 1 comment

    • No. I was referring to the "standard concentration bound" in that paper, which applies when you have separate validation and test sets. I think the argument can usually be improved by applying small-variance inequalities such as Bernstein's, to excess risk-like quantities such…

      iflpView on HNParent context

  2. A cartel of influential datasets are dominating machine learning research 1 comment

    • I disagree: some "collective overfitting" happens when everyone evaluates on the same dataset (and often, the same test set). There's a neat set of papers by Recht et al. showing results are slightly overfit to the test partitions of ImageNet and CIFAR-10: rotating examples…

      mattkrauseView on HNParent context

  3. Why neural networks struggle with the Game of Life 1 comment

    • Yes, because your validation set performance is imperfectly correlated with performance on more samples (because it's of finite size), so it will tend to overestimate within-distribution. There's also the problem that you probably want to deploy it on data which was not exactly…

      gwernView on HNParent context

  4. Do We Train on Test Data? Purging Cifar of Near-Duplicates 1 comment
  5. Training ResNet-50 on ImageNet in 35 Epochs Using Second-Order Optimization 1 comment

    • Yes, over-fitting is definitely something that happens. Here's a paper that demonstrates the problem by testing CIFAR classifiers on a new version of the input data: https://arxiv.org/abs/1806.00451 The flip side is that by having a common problem to tackle it's much easier to…

      asparaguiView on HNParent context

  6. Current state of the art in objects classification 1 comment

    • Many years of "human optimization" on the same old datasets must lead to severe overfitting, so these results should be taken with a grain of salt. Related research: https://arxiv.org/abs/1806.00451

      DzugaruView on HNParent context

  7. Building the Software 2.0 Stack by Andrej Karpathy [video] 1 comment

    • Treating training datasets as dynamic components of ML systems, along with the corresponding tooling and infrastructure, is one of the most under-appreciated points in the field today. Part of the cause, I think, is due to stigma coming from the academic side that dataset…

      dkislyukView on HNParent context

  8. Why do CNNs generalize so poorly to small image transformations? 1 comment

    • This paper is definitely quite interesting, but before this thread turns into a bunch of NN hate, here is another take: "Do CIFAR-10 Classifiers Generalize to CIFAR-10?" - https://arxiv.org/abs/1806.00451 They use the same procedure used to construct CIFAR-10 to construct a new…

      EridrusView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.