The Role of ImageNet Classes in Fréchet Inception Distance · HackerPapers discussions

The Role of ImageNet Classes in Fréchet Inception Distance — mentioned by godelski

Tue, 11 Jul 2023 22:23:18 GMT

> I have used LLMs on actually novel tasks for which there aren’t any datasets out there. They “get it”. Can you give an example so that we may better discuss or that I can adequately update my understandings? But I will say that simplifying this down to "just trained to predict the next token" is not accurate as it does not account for the differences in architectures and cost functions which dramatically affect this statement due to the differences in their biases. As a clear example, training an image model on likelihood does note guarantee that the model will produce high fidelity…

Paper and discussion history. A mention is not an endorsement.

The Role of ImageNet Classes in Fréchet Inception Distance — mentioned by godelski

Tue, 30 May 2023 19:46:35 GMT

Oh yeah, those are great papers, which I wish more people read. (I'll never not laugh at the journal name) I'll add this too[0] since it seems to be highly missed. But as far as validation sets, I have a hard time convincing anyone I talk to that tuning on test data is information leakage. I've also had issues with discussing people (even outside my lab, even at big schools/labs) what "uncurated samples" means. I say "you sample batch and show that" they say you can sample multiple batches and pick the best one... So this is the same kind of thinking and shows why evaluating generative papers…

Paper and discussion history. A mention is not an endorsement.

The Role of ImageNet Classes in Fréchet Inception Distance — mentioned by godelski

Sun, 09 Apr 2023 21:32:46 GMT

It is also important to remember that there are different inductive biases for different networks. Sutton's "Bitter Lesson"[0] argues for scale + more arbitrary complexity. ViTs scale really well because some of this arbitrary complexity. But I agree with your point that what to use is not obvious and this gets really complicated real quickly. For example, Swin loses some of the properties of self-attention but Neighborhood Attention doesn't[1] (these are both considered reductive attention types). Does this have a large effect? Might depend on your task. Looking at non-classification tasks…

Paper and discussion history. A mention is not an endorsement.