Improved Precision and Recall Metric for Assessing Generative Models · HackerPapers discussions

Improved Precision and Recall Metric for Assessing Generative Models — mentioned by godelski

Tue, 11 Jul 2023 22:23:18 GMT

> I have used LLMs on actually novel tasks for which there aren’t any datasets out there. They “get it”. Can you give an example so that we may better discuss or that I can adequately update my understandings? But I will say that simplifying this down to "just trained to predict the next token" is not accurate as it does not account for the differences in architectures and cost functions which dramatically affect this statement due to the differences in their biases. As a clear example, training an image model on likelihood does note guarantee that the model will produce high fidelity…

Paper and discussion history. A mention is not an endorsement.

Improved Precision and Recall Metric for Assessing Generative Models — mentioned by godelski

Tue, 30 May 2023 19:46:35 GMT

Oh yeah, those are great papers, which I wish more people read. (I'll never not laugh at the journal name) I'll add this too[0] since it seems to be highly missed. But as far as validation sets, I have a hard time convincing anyone I talk to that tuning on test data is information leakage. I've also had issues with discussing people (even outside my lab, even at big schools/labs) what "uncurated samples" means. I say "you sample batch and show that" they say you can sample multiple batches and pick the best one... So this is the same kind of thinking and shows why evaluating generative papers…

Paper and discussion history. A mention is not an endorsement.