Gain insights into basic and intermediate deep learning concepts, including CNNs, RNNs, GANs, and transformers. Delve into fundamental architectures to enhance your machine learning model training skills.

docker.tar.gz

Pytorch

GANS

This course is an accumulation of well-grounded knowledge and experience in deep learning. It provides you with the basic concepts you need in order to start working with and training various machine learning models. 

You will cover both basic and intermediate concepts including but not limited to: convolutional neural networks, recurrent neural networks, generative adversarial networks as well as transformers.

After completing this course, you will have a comprehensive understanding of the fundamental architectural components of deep learning. Whether you’re a data and computer scientist, computer and big data engineer, solution architect, or software engineer, you will benefit from this course.

Introduction to Deep Learning & Neural Networks

The decoder consists of all the aforementioned components plus two novel ones. As before:


1. The output sequence is fed in its entirety, and word embeddings are computed.

2. Positional encoding is again applied.

3. The vectors are passed to the first decoder block.


Each decoder block includes:



1. A **masked** multi-head self-attention layer 

2. A normalization layer followed by a residual connection

3. A new multi-head attention layer (known as **encoder-decoder attention**)

4. A second normalization layer and a residual connection

5. A linear layer and a third residual connection


The decoder block appears again $N=6$ repeated times. The final output is transformed through a final linear layer, and the output probabilities are calculated with the standard softmax function. 

The output probabilities predict the next token in the output sentence.

How?

For the machine translation example, we assign a probability to each word in the French language, and then, we simply keep the one with the highest score.

To put things into perspective, the original model was trained on a 2014 English-French dataset consisting of 36 M sentences and 32,000 tokens.

While you are familiar with most concepts of the decoder, there are two more that we need to discuss. Let’s start with the
 **masked** multi-head self-attention layer.


# Masked multi-head self-attention

In case you haven’t realized, in the decoding stage, we predict one word (token) after another. In NLP problems like machine translation, the sequential token prediction is unavoidable. As a result, the self-attention layer needs to be modified in order to consider only the output sentence that has been generated so far. 

In our translation example “Hello I love you”, the input of the decoder on the **third pass** will be “Bonjour”, “je … …”. 

> As you can tell, the difference here is that we don’t know the whole sentence because it hasn’t been produced yet. That is why we need to disregard the unknown words. 
Otherwise, the model would just copy the next word! To achieve this, we mask the next word embeddings (by setting them to $-\inf$). 

Mathematically, we have:

$$
MaskedAttention({Q}, {K}, {V})={softmax}\left(\frac{{Q} {K}^{T} + {M} }{\sqrt{d_{k}}}\right) {V}
$$

where the matrix M (mask) consists of zeros and $-\inf$. 

> Eventually, the zeros will become ones with the exponential, while infinities become zeros.

This effectively has the same effect as removing the corresponding connection. The remaining principles are exactly the same as the encoder’s attention. Once again, we can implement them in parallel to speed up the computations.

# Teacher forcing

Obviously, the mask will change for every new token that we compute. We will take the model's predicted word and use it as input to the sequence and try to predict the next one.

This kind of learning is called **teacher forcing** in the literature. We use the prediction of the model as input. This brings the model closer to the testing setup where we will necessarily have to use the predicted token (word) as input in the next step.



# Encoder-decoder attention: where the magic happens

> Encoder-decoder attention is simply the multi-head self-attention that we have already described. The difference is that the query $Q$ will come from a different source than the keys $K$ and values $V$.

You can also find it in the literature as **cross attention**. What is it?

This is actually where the decoder processes the **final** encoded representation. Remember that the encoder is something like our database. It will give us the final keys $K$ and values $V$.

Formulate the transformer's decoder and learn about masked multi-head self-attention.

Learn Deep Learning

Neural Networks

Training Neural Networks

Convolutional Neural Networks

Recurrent Neural Networks

Autoencoders

Generative Adversarial Networks

Attention and Transformers

Graph Neural Networks

Conclusion

Final Quiz

The Transformer's Decoder