Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks · HackerPapers discussions

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks — HN submission

Fri, 28 Jul 2023 11:50:37 GMT

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks — mentioned by anishathalye

Thu, 27 Jul 2023 15:32:15 GMT

Sharing some context here: in grad school, I spent months writing custom data analysis code and training ML models to find errors in large-scale datasets like ImageNet, work that eventually resulted in this paper (https://arxiv.org/abs/2103.14749) and demo (https://labelerrors.com/). Since then, I’ve been interested in building tools to automate this sort of analysis. We’ve finally gotten to the point where a web app can do automatically in a couple of hours what I spent months doing in Jupyter notebooks back in 2019—2020. It was really neat to see the software we built automatically produce…

Paper and discussion history. A mention is not an endorsement.

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks — mentioned by ckastner

Mon, 06 Dec 2021 13:23:43 GMT

Also interesting in this context: this recent paper [1] analyzed some of the most popular datasets and found numerous label errors in the test sets. For example, they estimated at least 6% of the samples in the ImageNet validation set were misclassified, with surprising consequences when comparing model performances using corrected sets. [1] Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, https://arxiv.org/pdf/2103.14749.pdf

Paper and discussion history. A mention is not an endorsement.

Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks — HN submission

Thu, 28 Oct 2021 15:04:40 GMT

Our datasets are flawed. ImageNet has an error rate of ~5.8%

23 points; 4 discussion replies on HN. Paper and source history. A submission is not an endorsement.