Character-Aware Models Improve Visual Text Rendering · HackerPapers discussions

Character-Aware Models Improve Visual Text Rendering — mentioned by gwern

Sun, 17 Mar 2024 18:51:13 GMT

> Ask DALL-E 3 to make a picture with some text, and it will spit out 4 image. Usually 2 or 3 are perfectly spelled. Yes, this is the 'miracle of spelling' (as https://arxiv.org/abs/2212.10562#google calls it): for many words, larger models can manage to deduce the spelling somehow despite the tokenization. It may even fool you into thinking it understands spelling in general. But if you ask DALL-E 3 to generate a random string of ASCII, you'll quickly discover the limits to the 'miracle'.

Paper and discussion history. A mention is not an endorsement.

Character-Aware Models Improve Visual Text Rendering — mentioned by gwern

Sat, 16 Mar 2024 20:08:39 GMT

Correct. Most image models to date have used very small text models, which are unable to spell many words (https://arxiv.org/abs/2108.11193) - if you cannot spell a word letter by letter because you don't know what the letters are, how are you going to generate a pixel image of each of those letters...? (It is not as if letters were intrinsically hard. GANs were generating letters easily like 7 years ago.) Their general inability is due to BPEs and non-character-based tokenization. This was noted in the DALL-E 2 paper (https://arxiv.org/pdf/2204.06125.pdf#page=16&org=openai), and it can be…

Paper and discussion history. A mention is not an endorsement.

Character-Aware Models Improve Visual Text Rendering — mentioned by ollin

Fri, 20 Oct 2023 04:47:51 GMT

"Character-Aware Models Improve Visual Text Rendering" https://arxiv.org/abs/2212.10562 for anyone curious

Paper and discussion history. A mention is not an endorsement.

Character-Aware Models Improve Visual Text Rendering — mentioned by radq

Fri, 28 Apr 2023 17:30:34 GMT

It is most likely due to the text encoder - see "Character-Aware Models Improve Visual Text Rendering". https://arxiv.org/abs/2212.10562

Paper and discussion history. A mention is not an endorsement.

Character-Aware Models Improve Visual Text Rendering — mentioned by woadwarrior01

Mon, 10 Apr 2023 07:49:38 GMT

> LLMs work on tokens, which, for the most part, are words (longer words are generally split up into multiple tokens). Although this is mostly true, there are exceptions to the rule. Look at the ByT5 family[1] of models for instance. These models operate on byte level representations of text and are inherently token free. I read a recent paper[2] that found that replacing token based text encoders in text to image models with these byte level models improves the spelling abilities of the txt2img models (which are otherwise notoriously bad at spelling in all human languages). [1]…

Paper and discussion history. A mention is not an endorsement.

Character-Aware Models Improve Visual Text Rendering — mentioned by ebalit

Thu, 16 Feb 2023 00:49:56 GMT

It's a side effect of the way the text input is represented before being used by the model. It doesn't get the text as a sequence of chars but as a sequence of tokens. This paper [1] shows that giving character-level awareness to the model can improve the "visual spelling". 1: https://arxiv.org/abs/2212.10562

Paper and discussion history. A mention is not an endorsement.

Character-Aware Models Improve Visual Text Rendering — mentioned by gwern

Mon, 26 Dec 2022 02:48:17 GMT

https://www.gwern.net/GPT-3#bpes for further reading. https://arxiv.org/abs/2212.10562#google is a particularly impressive recent demonstration of how serious the issue is.

Paper and discussion history. A mention is not an endorsement.

Character-Aware Models Improve Visual Text Rendering — HN submission

Thu, 22 Dec 2022 09:24:49 GMT

Improving Text Rendering in Image Generation Models

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.