Self-Attention vs. Convolution

Let's explore how we can employ self-attention in computer vision.

Comparing self-attention and convolution in computer vision

The process of generating self-attended feature maps involves a series of transformations applied to a 3D image representation, denoted as $X$. Initially, $X$ is a 2D image transformed by weight matrices $W$, with an added third dimension representing channel information, such as different colors.

The first step involves the extraction of three sets of weight matrices, namely $W_k$, $W_q$, and $W_v$. Subsequently, these weight matrices are applied to the original image representation $X$ to form three distinct matrices: $K$, $Q$, and $V$ ...