HackerPapers

arXiv:1511.01844stat.ML

A note on the evaluation of generative models

Lucas Theis, Aäron van den Oord, Matthias Bethge

Abstract

Probabilistic generative models can be used for compression, denoising, inpainting, texture synthesis, semi-supervised learning, unsupervised feature learning, and other tasks. Given this wide range of applications, it is not surprising that a lot of heterogeneity exists in the way these models are formulated, trained, and evaluated. As a consequence, direct comparison between models is often difficult. This article reviews mostly known but often underappreciated properties relating to the evaluation and interpretation of generative models with a focus on image models. In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional. Good performance with respect to one criterion therefore need not imply good performance with respect to the other criteria. Our results show that extrapolation from one criterion to another is not warranted and generative models need to be evaluated directly with respect to the application(s) they were intended for. In addition, we provide examples demonstrating that Parzen window estimates should generally be avoided.

arXiv comments
Journal reference
DOI
Subjects
stat.MLMachine Learning (Statistics) cs.LGMachine Learning

On Hacker News

Linked by 2 identifiable commenters in 5 comments across 5 threads, from Feb 2017 to Jul 2023.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Classifying customer messages with LLMs vs traditional ML 1 comment

    • > I have used LLMs on actually novel tasks for which there aren’t any datasets out there. They “get it”. Can you give an example so that we may better discuss or that I can adequately update my understandings? But I will say that simplifying this down to "just trained to predict…

      godelskiView on HNParent context

  2. Production AI systems are hard 1 comment

    • Oh yeah, those are great papers, which I wish more people read. (I'll never not laugh at the journal name) I'll add this too[0] since it seems to be highly missed. But as far as validation sets, I have a hard time convincing anyone I talk to that tuning on test data is…

      godelskiView on HNParent context

  3. Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows 1 comment

    • It is also important to remember that there are different inductive biases for different networks. Sutton's "Bitter Lesson"[0] argues for scale + more arbitrary complexity. ViTs scale really well because some of this arbitrary complexity. But I agree with your point that what to…

      godelskiView on HNParent context

  4. My Favorite Deep Learning Papers of 2017 1 comment

    • GANs are for graphics, not machine learning. They have no test scores as they cannot run on the test set. Therefore, its hard to tell how much they overfit. But they have uses, like for lossy compression of images or texture generation. Stuff which focuses on the graphical side…

      317070View on HNParent context

  5. AI learns to write its own code by stealing from other programs 1 comment

    • Actually, there is a formal method to evaluate generative models. All you need is a validation set. Then you need to find what the odds are that the model would have generated your validation set. That probability is the one you want to maximize [0]. In order to be able to…

      317070View on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.