TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up · HackerPapers discussions

TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up — mentioned by kromem

Sun, 20 Mar 2022 06:05:37 GMT

My whole point is that the way in which AI progress can intersect itself can lead to compounding effects in ways technology has not. One way to improve upon GPT-3 is almost certainly going to be adding discriminators into the mix: "GeDi: A Powerful New Method for Controlling Language Models" https://blog.salesforceairesearch.com/gedi/ And it may be that rather than it being a CNN discriminator such as in that case, it will be a second transformer: "[2102.07074] TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up" https://arxiv.org/abs/2102.07074 My point is we may…

Paper and discussion history. A mention is not an endorsement.

TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up — mentioned by unixpickle

Sat, 10 Jul 2021 21:23:24 GMT

I think you are conflating "Transformers" and "autoregressive models". Transformers are a general purpose architecture for transforming sequences into other sequences with self-attention. AR models / GANs are frameworks for generative modeling. The model architecture is almost entirely orthogonal to the generative framework. You can use transformers as part of GANs [1], and you can even use them as discriminative models for images [2]. [1] https://arxiv.org/abs/2102.07074 [2] https://arxiv.org/abs/2010.11929

Paper and discussion history. A mention is not an endorsement.