Creating Videos from Images

GANs can generate novel photorealistic images after being trained on a group of example photos. This technique can also be used to create variations of an image, either by applying “filters” or new poses or angles of the base image.

Limitations

Could we push this approach to its utmost potential and generate a talking head from just one or a few images? This task presents a significant challenge. Traditional methods, including deep learning techniques, often introduce noticeable artifacts when applying “warping” transformationsYaroslav Ganin, Daniil Kononenko, Diana Sungatullina, and Victor Lempitsky. 2016. “DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation.” ArXiv (Cornell University), July. https://doi.org/10.48550/arxiv.1607.07215. to a set of images, thereby diminishing the realism of the outputAverbuch-Elor, Hadar, Daniel Cohen-Or, Johannes Kopf, and Michael F. Cohen. 2017. “Bringing Portraits to Life.” ACM Transactions on Graphics 36 (6): 1–13. https://doi.org/10.1145/3130800.3130818..

Despite recent advances in convolutional neural networks (CNNs) capable of generating highly realistic human head images, these models typically necessitate training on extensive datasets portraying a single individual to establish a personalized talking head model. However, practical scenarios frequently require the creation of such models with limited input, possibly just a few image views or even a single image of an individual.

The synthesis of authentic talking head sequences poses considerable difficulty for two main reasons.

1. Human heads exhibit substantial photometric, geometric, and kinematic complexity, encompassing not only facial features but also aspects such as the mouth cavity, hair, and attire. This complexity presents challenges for modeling despite the existence of numerous approaches tailored to facial modeling.

2. The human visual system is remarkably sensitive to even minor errors in the appearance modeling of human heads, further complicating the task.

An alternative approach is to use generative models to sample potential angular and positional variations of the input images (shown in the figure “Generative architecture for creating moving frames from single images”), as performed by Zakharov et al. in their paper “Few-Shot Adversarial Learning of Realistic Neural Talking Head ModelsZakharov, Egor, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. 2019. “Few-Shot Adversarial Learning of Realistic Neural Talking Head Models,” May. https://doi.org/10.48550/arxiv.1905.08233..”

Few-shot learning

It is a system designed with few-shot capability that undergoes extensive meta-learning on a large dataset of videos. Subsequently, it leverages this meta-learning to facilitate few- and one-shot learning of neural talking head models for previously unseen individuals. This learning process is framed as adversarial training problems, employing high-capacity generators and discriminators.