Fantastic Generalization Measures and Where to Find Them · HackerPapers discussions

Fantastic Generalization Measures and Where to Find Them — mentioned by hodgehog11

Sat, 06 Sep 2025 00:02:08 GMT

Ugh, did they really have to continue on with that ridiculous title meme? For those that don't know, it's inspired by this paper: https://arxiv.org/abs/1912.02178 This may all seem in good fun, but it makes a real difference when you have to introduce this paper to students and other academics getting into the discipline. It's just embarrassing, and always garners a reaction from a facepalm to disgust. This is especially true now that Rowling is a controversial figure. As for the paper itself, this provides a good source for referencing, but the conclusions drawn here seem to be pretty…

Paper and discussion history. A mention is not an endorsement.

Fantastic Generalization Measures and Where to Find Them — mentioned by choppaface

Sun, 21 Mar 2021 19:21:33 GMT

But who actually uses your ImageNet DogBird dataset in an existing study? Part of the problem is that you're inventing your own benchmark. And yet you could have used SVHN and/or the same fixtures (i.e. datasets and ablations) of some of the work you cite e.g. as in "Fantastic Generalization Measures and Where to Find Them" https://arxiv.org/pdf/1912.02178.pdf The CIFAR-5m result is interesting, but it's misleading to the reader since the dataset is so contrived. And yet you lead with it on your front page. There's wayyyy too much hype going on here.

Paper and discussion history. A mention is not an endorsement.