https://hackerpaper.kaipfister.de/p/1502.04585/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMThttps://news.ycombinator.com/item?id=10731974urn:hackerpapers:mention:1502.04585:10731974Mon, 14 Dec 2015 16:31:57 GMT<p>Nice article on why it is cheating (from a mathematical sense, independent of language) to get scores from the hold-out leader board too many times (plus some methods to mitigate the effect): http://arxiv.org/abs/1502.04585 .</p><p><a href="https://hackerpaper.kaipfister.de/p/1502.04585/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.LGhttps://news.ycombinator.com/item?id=9660769urn:hackerpapers:mention:1502.04585:9660769Thu, 04 Jun 2015 17:36:51 GMT<p>This doesn&#x27;t affect the actual contest results on the test set. Anyone who wants to overfit to the validation set can do so, getting a nice temporary leaderboard spot at the price of test set generalization, which is what determines the winners. That said it&#x27;d obviously be nice to build leaderboards that are robust to this type of gaming. There&#x27;s been some recent work, mostly driven by Moritz Hardt AFAIK (briefly referenced in the submission), using ideas from differential privacy to create leaderboards that leak only a bounded amount of information about the validation set…</p><p><a href="https://hackerpaper.kaipfister.de/p/1502.04585/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.LG