Self-Attention Matrix Equations

The self-attention mechanism plays a crucial role in the architecture of transformer models, enabling them to capture relationships between different elements in a sequence. In this lesson, we’ll study the intricacies of self-attention, starting with a practical example represented by a matrix equation. This example will serve as a foundation for understanding the inner workings of self-attention.

Introduction to the self-attention mechanism

Let's start by examining an example that illustrates the self-attention mechanism using a matrix equation. This will help us understand the internal workings of the process.

Encoding input vectors

Imagine we have a three-word sentence in French, "Je suis heureux", which we'll denote as x1, x2, and x3 to represent the word vectors for each word.

The role of query, key, and value projections

We aim to pass these words through a self-attention layer, which, as mentioned earlier, captures the relationships between words using self-attention.

In this case, we'll use $Q$, $K$, and $V$ as projections of the same input, $X$, with each having a different weight matrix. Each matrix should have the same number of rows as there are features in each embedding. When we multiply each word by these matrices, we get projections or views for the query, key, and value of the same word. So, while we previously stated that $Q$ equals $K$ equals $V$, they aren't exactly equal but rather derived from the same source using learnable weight matrices to represent the word as a query, key, and value.

The reason for this projection is twofold. First, it enables a learnable attention mechanism, going beyond mere semantic similarity, as previously discussed. A simple dot product represents semantic similarity, but with these weight matrices, we can incorporate various perspectives of the input word, offering different features. This leads us to the second benefit, which is having multiple views of the same word, such as part-of-speech tags, named entities, or other learnable attributes. In summary, the three matrices, $W_q$, $W_k$ and $W_v$ ...