Understanding the Self-Attention Mechanism

How can we create the query, key, and value matrices? To create these, we introduce three new weight matrices called $W^Q$ , $W^K$ , $W^V$. We create the query, $Q$, key, $K$, and value, $V$, matrices by multiplying the input matrix $X$ by $W^Q$ , $W^K$ , and $W^V$ respectively.

As shown in the following figure, multiplying the input matrix ,$X$, by the weight matrices $W^Q$ , $W^K$ , and $W^V$, we obtain the query, key, and value matrices:

From the preceding figure, we can understand the following:

• The first row in the query, key, and value matrices—$q_1$ , $k_1$ , and $v_1$—implies the query, key, and value vectors of the word 'I'.

• The second row in the query, key, and value matrices— $q_2$ , $k_2$ , and $v_2$ —implies the query, key, and value vectors of the word 'am'.

• The third row in the query, key, and value matrices— $q_3$ , $k_3$ , and $v_3$ —implies the query, key, and value vectors of the word 'good'.

Dimensions of the query, key, and value matrices

Note that the dimensionality of the query, key, and value vectors is 64. Thus, the dimension of our query, key, and value matrices is:

Since we have three words in the sentence, the dimensions of the query, key, and value matrices are:

But still, the ultimate question is, why are we computing this? What is the use of query, key, and value matrices? How is this going to help us? This is exactly what we will discuss in detail in the next section.

How the self-attention mechanism works

We learned how to compute query, $Q$, key, $K$, and value, $V$, matrices and we also learned that they are obtained from the input matrix, $X$. Now, let's see how the query, key, and value matrices are used in the self-attention mechanism.

We learned that in order to compute a representation of a word, the self-attention mechanism relates the word to all the words in the given sentence. Consider the sentence 'I am good'. To compute the representation of the word 'I', we relate the word 'I' to all the words in the sentence, as shown in the following figure:

But why do we need to do this? Understanding how a word is related to all the words in the sentence helps us to learn better representation. Now, let's learn how the self-attention mechanism relates a word to all the words in the sentence using the query, key, and value matrices. The self-attention mechanism ...