Hierarchical Text-Conditional Image Generation with CLIP Latents · HackerPapers discussions

Hierarchical Text-Conditional Image Generation with CLIP Latents — mentioned by gwern

Sat, 16 Mar 2024 20:08:39 GMT

Correct. Most image models to date have used very small text models, which are unable to spell many words (https://arxiv.org/abs/2108.11193) - if you cannot spell a word letter by letter because you don't know what the letters are, how are you going to generate a pixel image of each of those letters...? (It is not as if letters were intrinsically hard. GANs were generating letters easily like 7 years ago.) Their general inability is due to BPEs and non-character-based tokenization. This was noted in the DALL-E 2 paper (https://arxiv.org/pdf/2204.06125.pdf#page=16&org=openai), and it can be…

Paper and discussion history. A mention is not an endorsement.

Hierarchical Text-Conditional Image Generation with CLIP Latents — mentioned by benlivengood

Mon, 02 May 2022 17:44:53 GMT

The original paper examines some likely causes for failure to encode relational details between subjects, e.g. the corgi with a red hat and green necktie or vice versa chosen seemingly at random. The model seems to learn about contrasting subjects but not track particularly which attributes belong to which subjects. This is, as far as I can tell from the original paper, inherent in the training and would require different data sets or more parameters to learn a greater number of subject-attribute associations in each image. https://arxiv.org/abs/2204.06125

Paper and discussion history. A mention is not an endorsement.