HackerPapers

arXiv:2212.10562cs.CL

Character-Aware Models Improve Visual Text Rendering

Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, Noah Constant

Abstract

Current image generation models struggle to reliably produce well-formed visual text. In this paper, we investigate a key contributing factor: popular text-to-image models lack character-level input features, making it much harder to predict a word's visual makeup as a series of glyphs. To quantify this effect, we conduct a series of experiments comparing character-aware vs. character-blind text encoders. In the text-only domain, we find that character-aware models provide large gains on a novel spelling task (WikiSpell). Applying our learnings to the visual domain, we train a suite of image generation models, and show that character-aware variants outperform their character-blind counterparts across a range of novel text rendering tasks (our DrawText benchmark). Our models set a much higher state-of-the-art on visual spelling, with 30+ point accuracy gains over competitors on rare words, despite training on far fewer examples.

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.CVComputer Vision and Pattern Recognition

On Hacker News

Linked by 5 identifiable commenters in 7 comments across 6 threads, from Dec 2022 to Mar 2024. Found in 1 HN submission from Dec 2022 to Dec 2022. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments6 threads

  1. Ask HN: Why can't image generation models spell? 2 comments

    • Correct. Most image models to date have used very small text models, which are unable to spell many words (https://arxiv.org/abs/2108.11193) - if you cannot spell a word letter by letter because you don't know what the letters are, how are you going to generate a pixel image of…

      gwernView on HNParent context

    • > Ask DALL-E 3 to make a picture with some text, and it will spit out 4 image. Usually 2 or 3 are perfectly spelled. Yes, this is the 'miracle of spelling' (as https://arxiv.org/abs/2212.10562#google calls it): for many words, larger models can manage to deduce the spelling…

      gwernView on HNParent context

  2. Improving image generation with better captions [pdf] 1 comment

    • "Character-Aware Models Improve Visual Text Rendering" https://arxiv.org/abs/2212.10562 for anyone curious

      ollinView on HNParent context

  3. DeepFloyd IF: open-source text-to-image model 1 comment

    • It is most likely due to the text encoder - see "Character-Aware Models Improve Visual Text Rendering". https://arxiv.org/abs/2212.10562

      radqView on HNParent context

  4. A Baby GPT 1 comment

    • > LLMs work on tokens, which, for the most part, are words (longer words are generally split up into multiple tokens). Although this is mostly true, there are exceptions to the rule. Look at the ByT5 family[1] of models for instance. These models operate on byte level…

      woadwarrior01View on HNParent context

  5. Illusion Diffusion: Optical Illusions Using Stable Diffusion 1 comment

    • It's a side effect of the way the text input is represented before being used by the model. It doesn't get the text as a sequence of chars but as a sequence of tokens. This paper [1] shows that giving character-level awareness to the model can improve the "visual spelling". 1…

      ebalitView on HNParent context

  6. What ChatGPT can't do 1 comment

    • https://www.gwern.net/GPT-3#bpes for further reading. https://arxiv.org/abs/2212.10562#google is a particularly impressive recent demonstration of how serious the issue is.

      gwernView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.