Face tracking in Unity

Face tracking refers to the ability of the AR system to detect and track a user’s face in real time using the device’s camera. It allows you to create interactive augmented reality experiences that are specifically tied to facial expressions and movements.

Implement face tracking

Face tracking technology can recognize facial features and expressions, such as the position and orientation of the eyes, nose, mouth, and head. Here is how you can implement it. Create a new AR project in Unity.

AR face manager component

The AR face manager is a component provided by the AR foundation framework that facilitates face tracking functionality in augmented reality applications. It enables you to detect and track the user's face in real-time using the device's front-facing camera. In XR origin, add the AR face manager component. You can change the maximum face count to the number of faces you want to track.

AR default face

The AR default face is a default GameObject pre-made by Unity. We will use as a prefab for the AR face manager. In the hierarchy tab, create > XR > AR default faceIt renders a default material on detected face meshes.. Drag it into the prefabs folder. From there, drag the AR default face from the prefabs folder to the face prefab in XR origin.

Main camera

In XR origin, double-click on main camera and make the following changes:

Clipping planes

The clipping planes property of the main camera specifies the distance range within which objects will be rendered. It consists of two values: the "Near" value and the "Far" value. The "Near" value determines the closest distance from the camera where objects will start rendering, and the "Far" value determines the maximum distance up to which objects will be visible. In this case, change the "Far" value to 1000 units. Objects beyond this distance won't be rendered, and they will appear clipped from the scene.

Depth

The depth property determines the rendering priority of the camera compared to other cameras in the scene. Chnage the depth value to -1, which indicates that this camera will be rendered first in the scene, ensuring the AR content is properly composited with the real-world camera feed and overlays correctly.

Facing direction

The AR camera manager is a component in Unity's AR foundation framework that manages the AR camera's functionality and settings. It provides an interface to the device's camera and handles AR session configurations. Under the AR camera manager, you can find the main camera, which is automatically set as the primary camera for AR rendering. The AR camera manager simplifies the setup and configuration of the AR camera, making it easier for you to create AR applications without dealing with low-level camera settings. Change the facing direction to "user." By setting the face direction to the user, it'll open the front camera of your device.

Save the scene. Connect your Android or iOS device. Go to file > build and run.

Demonstration

Conclusion

Face tracking opens up many possibilities for creating entertaining and engaging AR experiences, such as face filters similar to those popularized by social media platforms.