Local vs. Global Attention

We've previously explored global attention mechanisms, which establish connections across all inputs—be they spatial, channel-related, or temporal. Now, let's explore another critical aspect: local attention.

Local attention mechanism

As known, convolution is a local operation, due to its inductive bias or modeling assumption, while attention was identified as global, devoid of modeling assumptions, or low in inductive bias. Spatial attention, as depicted, links each blue pixel in space to a red pixel, capturing their relationship through an attention map. This is known as non-local attention, although other options are available.