HackerPapers

arXiv:2205.15868cs.CV

CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, Jie Tang

Early discovery. Linked by one identifiable commenter. This paper is outside the main ranking; a mention is not an endorsement.

Abstract

Large-scale pretrained transformers have created milestones in text (GPT-3) and text-to-image (DALL-E and CogView) generation. Its application to video generation is still facing many challenges: The potential huge computation cost makes the training from scratch unaffordable; The scarcity and weak relevance of text-video datasets hinder the model understanding complex movement semantics. In this work, we present 9B-parameter transformer CogVideo, trained by inheriting a pretrained text-to-image model, CogView2. We also propose multi-frame-rate hierarchical training strategy to better align text and video clips. As (probably) the first open-source large-scale pretrained text-to-video model, CogVideo outperforms all publicly available models at a large margin in machine and human evaluations.

arXiv comments
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 1 identifiable commenter in 1 comment across 1 thread, from Sep 2022 to Sep 2022.

Comments linking this paper, per year.

Discussed in comments1 thread

  1. AI Content Generation, Part 1: Machine Learning Basics 1 comment

    • Depending on how you're measuring it, video generation will come in less than 12 months, or it's already here. - In the Video Diffusion paper[1] they generate 64x64 16 frame videos, model is not released, but an open source implementation using an imagen-like pipeline exists[2]…

      nodjaView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.