A note on the evaluation of generative models · HackerPapers discussions

A note on the evaluation of generative models — mentioned by godelski

Tue, 11 Jul 2023 22:23:18 GMT

> I have used LLMs on actually novel tasks for which there aren’t any datasets out there. They “get it”. Can you give an example so that we may better discuss or that I can adequately update my understandings? But I will say that simplifying this down to "just trained to predict the next token" is not accurate as it does not account for the differences in architectures and cost functions which dramatically affect this statement due to the differences in their biases. As a clear example, training an image model on likelihood does note guarantee that the model will produce high fidelity…

Paper and discussion history. A mention is not an endorsement.

A note on the evaluation of generative models — mentioned by godelski

Tue, 30 May 2023 19:46:35 GMT

Oh yeah, those are great papers, which I wish more people read. (I'll never not laugh at the journal name) I'll add this too[0] since it seems to be highly missed. But as far as validation sets, I have a hard time convincing anyone I talk to that tuning on test data is information leakage. I've also had issues with discussing people (even outside my lab, even at big schools/labs) what "uncurated samples" means. I say "you sample batch and show that" they say you can sample multiple batches and pick the best one... So this is the same kind of thinking and shows why evaluating generative papers…

Paper and discussion history. A mention is not an endorsement.

A note on the evaluation of generative models — mentioned by godelski

Sun, 09 Apr 2023 21:32:46 GMT

It is also important to remember that there are different inductive biases for different networks. Sutton's "Bitter Lesson"[0] argues for scale + more arbitrary complexity. ViTs scale really well because some of this arbitrary complexity. But I agree with your point that what to use is not obvious and this gets really complicated real quickly. For example, Swin loses some of the properties of self-attention but Neighborhood Attention doesn't[1] (these are both considered reductive attention types). Does this have a large effect? Might depend on your task. Looking at non-classification tasks…

Paper and discussion history. A mention is not an endorsement.

A note on the evaluation of generative models — mentioned by 317070

Thu, 28 Dec 2017 08:06:12 GMT

GANs are for graphics, not machine learning. They have no test scores as they cannot run on the test set. Therefore, its hard to tell how much they overfit. But they have uses, like for lossy compression of images or texture generation. Stuff which focuses on the graphical side of things where it outperforms machine learning methods with its crisper samples. For an overview of this argument: https://arxiv.org/abs/1511.01844 The idea of adversarial training is important and relevant in ML though! It allows for setting up losses which are hard to formulate otherwise.

Paper and discussion history. A mention is not an endorsement.

A note on the evaluation of generative models — mentioned by 317070

Thu, 23 Feb 2017 10:08:49 GMT

Actually, there is a formal method to evaluate generative models. All you need is a validation set. Then you need to find what the odds are that the model would have generated your validation set. That probability is the one you want to maximize [0]. In order to be able to calculate or estimate this probability, you need to make some amendments to your model architecture. But for instance PixelCNN's know the probability exactly, variational auto-encoders have a lower bound on the probability, etc. It is mainly the generative adverserial networks (GANs) which cannot. And this is the reason…

Paper and discussion history. A mention is not an endorsement.