HackerPapers

arXiv:2203.13131cs.CV

Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors

Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, Yaniv Taigman

Abstract

Recent text-to-image generation methods provide a simple yet exciting conversion capability between text and image domains. While these methods have incrementally improved the generated image fidelity and text relevancy, several pivotal gaps remain unanswered, limiting applicability and quality. We propose a novel text-to-image method that addresses these gaps by (i) enabling a simple control mechanism complementary to text in the form of a scene, (ii) introducing elements that substantially improve the tokenization process by employing domain-specific knowledge over key image regions (faces and salient objects), and (iii) adapting classifier-free guidance for the transformer use case. Our model achieves state-of-the-art FID and human evaluation results, unlocking the ability to generate high fidelity images in a resolution of 512x512 pixels, significantly improving visual quality. Through scene controllability, we introduce several new capabilities: (i) Scene editing, (ii) text editing with anchor scenes, (iii) overcoming out-of-distribution text prompts, and (iv) story illustration generation, as demonstrated in the story we wrote.

arXiv comments
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.AIArtificial Intelligence cs.CLComputation and Language cs.GRGraphics cs.LGMachine Learning

On Hacker News

Linked by 5 identifiable commenters in 8 comments across 7 threads, from Mar 2022 to Jul 2022. Found in 1 HN submission from Mar 2022 to Mar 2022. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments7 threads

  1. AI image generation: more creative control 1 comment

    • Blog post: https://ai.facebook.com/blog/greater-creative-control-for-ai... Paper: https://arxiv.org/abs/2203.13131 Meta AI's tweet: https://twitter.com/MetaAI/status/1547598454922153985?s=20&t...

      deviparikhView on HNParent context

  2. Meta's Make-a-Scene with AI = DALL-E with creative control via simple sketch 1 comment

    • More examples here https://www.instagram.com/p/Cf_tjL_rgEe/?hl=en I understand this is probably just a frontend to this paper from March https://arxiv.org/pdf/2203.13131.pdf Given the spotlight DALL-E has received makes sense to capitalise

      ht96View on HNParent context

  3. My deepfake DALL-E 2 vacation photos passed the Turing Test 1 comment

    • Facebook's Make-A-Scene, which looks almost as good as DALL-E 2, does something very much like that, using face-specific targeting to boost the quality of that part of images: https://arxiv.org/abs/2203.13131#facebook

      gwernView on HNParent context

  4. A few things to try with DALL·E 1 comment

    • > What is the content of the training set like, that DALL·E 2 has all this info? Web crawling. (https://arxiv.org/pdf/2103.00020.pdf 2.2) > Are there people out there spending thousands of hours just tagging or writing descriptions of training images? "Yes", Google Image Search…

      astrangeView on HNParent context

  5. Playing with DALL-E 2 2 comments

    • It's the PR risk. You can easily pass the 'look good to an uniformed observer' test with human faces. Remember, faces were something GANs were doing near-flawlessly back as far as ProGAN all the way back in the dark ages of late 2017. (Then StyleGAN did non-photograph faces…

      gwernView on HNParent context

    • Also Make-A-Scene, which in some ways is noticeably better than DALL-E 2 (faces, editing & control of layout through semantic segmentation conditioning): https://arxiv.org/abs/2203.13131#facebook

      gwernView on HNParent context

  6. Dall-E 2 1 comment

    • As I keep telling people: "hands are hard". This is why I went so far as to make a hand-specific dataset ("PALM" https://www.gwern.net/Crops#palm which of course now everyone is going to confuse with 'PaLM'...). Hands are just way too variable to learn easily. My dataset is a…

      gwernView on HNParent context

  7. Make-a-Scene: Scene-Based Text-to-Image Generation with Human Priors 1 comment

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.