Spatial vs. Channel vs. Temporal Attention

Understand the differences between spatial, channel, and temporal attention mechanisms used in vision transformers. Learn how each attention type processes image and video data to capture relationships across pixels, channels, and time frames. Gain practical insights including a basic implementation of spatial self-attention to enhance feature maps for computer vision tasks.

We'll cover the following...

Spatial attention
- Self-attention feature map generation
Channel attention
- CBAM: Convolutional Block Attention Module
Temporal attention
- Extension to temporal attention
A simple code implementation

Let's discuss the differences between channel, spatial, and temporal attention mechanisms.

Spatial attention

When working with input feature maps of size $N \cdot H \cdot W$ , spatial attention focuses on aggregating pixels $(H \cdot W)$ using self-attention. The resulting attention map has a size of $HW \cdot HW$ , capturing relations across all pixels.

Self-attention feature map generation

Now, consider self-attention feature map generation. Starting with a 3D image tensor $X$ , where dimensions $N$ , $H$ , and $W$ represent the number of colors, height, and width of either the input image or an intermediate feature map, we can attend to all the two-dimensional spatial relations $(H \cdot W)$ ...

1.Introduction

2.Overview of Transformer Networks

Mini Project

3.Transformers in Computer Vision

Project

4.Transformers in Image Classification

Mini Project

5.Transformers in Object Detection

6.Transformers in Semantic Segmentation

7.Spatio-Temporal Transformers

Mini Project

8.Wrap Up

Mock Interview

Spatial vs. Channel vs. Temporal Attention

Spatial attention

Self-attention feature map generation