Cold Case: The Lost MNIST Digits · HackerPapers discussions

Cold Case: The Lost MNIST Digits — mentioned by mattkrause

Mon, 06 Dec 2021 17:10:29 GMT

I disagree: some "collective overfitting" happens when everyone evaluates on the same dataset (and often, the same test set). There's a neat set of papers by Recht et al. showing results are slightly overfit to the test partitions of ImageNet and CIFAR-10: rotating examples between the train and test partitions causes systems to perform up to 10-15% worse. https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811 There's another neat bit of work involving MNIST. The original dataset (from the mid-90s) had 60,000 test examples, but the distributed versions that virtually everyone uses…

Paper and discussion history. A mention is not an endorsement.

Cold Case: The Lost MNIST Digits — mentioned by gwern

Thu, 24 Sep 2020 15:16:35 GMT

Yes, because your validation set performance is imperfectly correlated with performance on more samples (because it's of finite size), so it will tend to overestimate within-distribution. There's also the problem that you probably want to deploy it on data which was not exactly collected the same way. So you have issues of both internal and external validity. In practice, at least for CNN classifiers on ImageNet etc, it seems to be a pretty minor issue overall. Some links: https://arxiv.org/abs/1711.11561 https://arxiv.org/abs/1806.00451 https://arxiv.org/abs/1902.10811…

Paper and discussion history. A mention is not an endorsement.

Cold Case: The Lost MNIST Digits — HN submission

Sat, 01 Jun 2019 16:41:36 GMT

Cold Case: The Lost Mnist Digits

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.