Shifter Window (Swin) Transformers

Let's explore an essential architecture that modifies the vision transformer or video concept to overcome the quadratic complexity issue associated with image patches and growing image sizes.

Quadratic complexity challenge

The quadratic complexity challenge emerges when employing a fixed 16 by 16 pixel patch in the width architecture. This design choice, while initially effective for handling image data, becomes a computational bottleneck as the image size expands. Consequently, this quadratic growth becomes impractical for large-scale images, imposing severe constraints on processing resources and hindering the efficiency of the vision transformer.

In essence, the fixed 16 by 16 pixel patch, while providing a structured approach to analyze image data, becomes a limiting factor when confronted with the demands of scalability. This challenge prompts the exploration of alternative architectures, leading to the introduction of the shifted window (Swin) transformersVision transformer, employs shifted windows for efficient computation, excelling in image recognition tasks., which adeptly address these computational complexities through innovative strategies such as variable patches and window division.

Swin architecture overview

Swin introduces two clever tricks to address these challenges: variable patches and window division.

Variable patches

The Swin architecture introduces variable patches, a notable departure from the fixed 16 by 16 pixel patch paradigm. Unlike the rigid approach of the width architecture, Swin affords each layer the flexibility to accommodate a changing number of image patches.

The idea behind variable patches in Swin is akin to the encoder's max-pool strategy, where the reduction in patch numbers aligns with the layer's progression. In the initial layers, a larger number of smaller patches are employed, ensuring a fine-grained analysis of local features. This mirrors the early stages of the ...