Spatio-Temporal Transformers

Let's explore the integration of transformers in tasks involving temporal relations, such as video processing and multiple frames of images. Transformers, initially designed for natural language processing (NLP), naturally extend to model temporal sequences, making them suitable for video analysis applications.

Spatial and temporal relations in video analysis

Building on spatial relations using self-attention mechanisms, transformers now address temporal aspects in video analysis. The dimensions shift from $(C, W, H)$(representing height, width, and channels) to $(T . H . W . C)$, introducing a time dimension. This enables modeling both spatial and temporal relations crucial for applications like moving object detection.

Video transformer network architecture

The Video Transformer Network (VTN) architecture was introduced in 2021. The main goal of this architecture is to perform video classification tasks.

This architecture processes each video frame with a convolutional neural network (CNN) backbone, extracting 2D embeddings, $f(x)$. These embeddings are then combined with positional embeddings$(PE_0, PE_1,\ldots ,PE_n)$, representing the time frames. A transformer encoder performs temporal attention across all frames, facilitating video classification. The final output involves a classification token attending over all time steps.