Text-Guided Video Generation Products

Text-to-video generation

More recently, text-to-video generation is reaching its climax, where companies like Runway Research have launched impactful text-guided video generation products, such as Gen-2. These models have similar architectures as the text-to-image model seen before, with a text encoder, image encoder, and video decoder. These models are capable of generating sequences of frames or entire videos. In the code example below, we take a pre-trained text-to-video model and apply “flying batman” as a prompt. The result shows a snippet of sequences of frames from the generated video.