Key Feature Set

The human face and body are key entities in this task of fake content generation. While deep learning architectures usually do not require hand-crafted features, a little nudge goes a long way when complex entities are involved. Particularly when dealing with the human face, apart from detecting the overall face in a given image or video, a deepfake solution also needs to focus on the eyes, mouth, and other features.

In this lesson, we’ll briefly cover a few important features leveraged by different deepfake solutions. These are as follows:

• Facial Action Coding System (FACS)

• 3D Morphable Model (3DMM)

• Facial landmarks

We will also undertake a couple of hands-on exercises to better understand these feature sets.

Facial Action Coding System (FACS)

Developed by Carl-Herman Hjortsjö in 1969 and later adopted and refined by Ekamn et al. in 1978, the Facial Action Coding System, or FACS, is an anatomy-based system for understanding facial movements. It is one of the most extensive and accurate coding systems for analyzing facial muscles to understand expressions and emotions.

The figure below depicts a few specific muscle actions and their associated meanings.

FACS consists of a detailed manual that human coders use to code each facial expression manually. The muscular activities are grouped into what are called Action Units, or AUs. These AUs represent muscular activities corresponding to facial expressions. The figure above describes a few sample AUs, pointing to the movement of eyebrows, lips, and other parts of the face.

Although the original FACS system required human coders, there are automated systems now available to computationally determine the correct AUs. Works such as the following leverage automated AUs to generate realistic results:

• GANimation: Anatomically-aware facial animation from a single ...