Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models · HackerPapers discussions

Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models — mentioned by godelski

Tue, 06 Aug 2024 08:26:48 GMT

> The rule in ML There's definitely attempts to revive things (in the general sense, not just GANs), but most successes appear to come from large labs dedicating equal computational resources to the older models and often by changing names. This can make things more confusing and make things appear to be changing faster, but once you can see this, you'll have an easier time keeping up (so being new, watch out for this). I'll give some examples that are easier to read[0-2] (i.e. don't need expert knowledge to understand the nuances). As an insider (ML researcher), my complaint isn't so much…

Paper and discussion history. A mention is not an endorsement.

Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models — mentioned by stevenwalton

Fri, 19 Jul 2024 19:04:04 GMT

> But the constraint that both eyes should have consistent reflection patterns is just another statistical regularity that appears in real photographs Hi, author here of a model that does really good on this[0]. My model is SOTA and has undergone a third party user study that shows it generates convincing images of faces[1]. AND my undergrad is in physics. I'm not saying this to brag, I'm giving my credentials. That I have deep knowledge in both generating realistic human faces and in physics. I've seen hundreds of thousands of generated faces from many different models and architectures. I…

Paper and discussion history. A mention is not an endorsement.

Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models — mentioned by godelski

Fri, 15 Dec 2023 19:31:04 GMT

Look, every single message I've said that you're trading quality for speed. That does in fact mean GANs are worse. I'm not sure why you think I have said anything short of that, I've explicitly agreed with you that diffusion produces higher quality images, and I'm not interested in repeating myself any further. We're talking about a fucking raspberry pi, it is definitely reasonable to want to __trade quality for speed__ when you're working on a tiny computer. I'm glad you're aware that FID has limits. For some reason this is uncommon. But there are more limits than the backbone classifier.…

Paper and discussion history. A mention is not an endorsement.