Gain insights into basic and intermediate deep learning concepts, including CNNs, RNNs, GANs, and transformers. Delve into fundamental architectures to enhance your machine learning model training skills.

docker.tar.gz

Pytorch

GANS

This course is an accumulation of well-grounded knowledge and experience in deep learning. It provides you with the basic concepts you need in order to start working with and training various machine learning models. 

You will cover both basic and intermediate concepts including but not limited to: convolutional neural networks, recurrent neural networks, generative adversarial networks as well as transformers.

After completing this course, you will have a comprehensive understanding of the fundamental architectural components of deep learning. Whether you’re a data and computer scientist, computer and big data engineer, solution architect, or software engineer, you will benefit from this course.

Introduction to Deep Learning & Neural Networks

It’s finally time to apply all we have learned about Transformers. The best way to do it is to build a Transformer Encoder from scratch. We will start by developing all the subcomponents and in the end, we will combine them to form the encoder. Let’s start with something simple.

> Disclaimer: Pytorch has its own built-in Transformer and attention modules. However, we believe that you can get a solid understanding only if you develop them yourself.


# Linear layers
A good first step is to build the linear subcomponent. A 2-layered feedforward network followed by dropout is good enough. So here is what the forward pass should look like:

1) Linear Layer
2) RELU as an activation function
3) Dropout
4) 2nd Linear layer

You can implement this yourself. Jump to the code below, and finish the `FeedForward` module. Note that this is not an exercise. It is only intended to solidify your understanding by revisiting how to build simple Pytorch modules.



import torch
import torch.nn as nn
import torch.nn.functional as F


class FeedForward(nn.Module):
    def __init__(self, d_model: int, d_ff: int, dropout: float = 0.1):
        """
        Args:
            `d_model`: model dimension
            `d_ff`: hidden dimension of feed forward layer
            `dropout`: ropout rate, default 0.1
        """
        super(FeedForward, self).__init__() 
        ## 1. DEFINE 2 LINEAR LAYERS AND DROPOUT HERE
       
       

    def forward(self, x: torch.FloatTensor) -> torch.FloatTensor:
        """
        Args:
            `x`: shape (batch_size, max_len, d_model)
        Returns:
            same shape as input x
        """
        ## 2.  RETURN THE FORWARD PASS 

FeedForward(10, 100, 0.1)

# Layer normalization

As a next step, let’s define a layer normalization module. To keep you from being overwhelmed, you will be provided the code for this one. Remember that a layer normalization module is defined as:

$$ LN(x) = \alpha(\frac{x - \mu(x) }{\sigma(x)}) + \beta$$ 

where $\mu$ is the mean and $\sigma$ the standard deviation of the input vector.

```python
class LayerNorm(nn.Module):
    def __init__(self, features: int, eps: float = 1e-6):
        # features = d_model
        super(LayerNorm, self).__init__()
        self.a = nn.Parameter(torch.ones(features))
        self.b = nn.Parameter(torch.zeros(features))
        self.eps = eps

    def forward(self, x: torch.FloatTensor) -> torch.FloatTensor:
        mean = x.mean(-1, keepdim=True)
        std = x.std(-1, keepdim=True)
        return self.a * (x - mean) / (std + self.eps) + self.b
```

# Skip connection

Next in line, we have residual connections. Once again, you will be given the code so you can focus on the actual Transformer modules.

```python

class SkipConnection(nn.Module):
    """
    A residual connection followed by a layer norm.
    Note for code simplicity the norm is first as opposed to last.
    """

    def __init__(self, size: int, dropout: float):
        super(SkipConnection, self).__init__()
        self.norm = LayerNorm(size)
        self.dropout = nn.Dropout(dropout)

    def forward(self, 
                x: torch.FloatTensor, 
                sublayer: Union[MultiHeadAttention, FeedForward]
                ) -> torch.FloatTensor:
        """Apply residual connection to any sublayer with the same size."""
        return x + self.dropout(sublayer(self.norm(x)))
```

Note that in the code,  our skip connection module can be applied either on a feedforward module or a multi-head attention module. Speaking of attention:

# Multihead attention

We have already seen the code on multi-head attention. `MultiHeadAttention` from the previous lesson can be used intact here. There is nothing more for us to do here. Let’s continue.


# Encoder

And now for the good stuff. It's finally time to combine all the previous classes and form the Transformer's encoder. This is where you come into play. In the following playground, there are 3 different modules.

## EncoderLayer

The `EncoderLayer` consists of the self-attention module and a feed-forward network. In both of those, a skip connection is applied.

## Encoder

The `Encoder` module is nothing more than a stack of N encoder layers followed by  layer normalization. This should be quite straightforward.

>Tip: To stack a list of modules, a neat trick is to use `nn.ModuleList` such as `nn.ModuleList([copy.deepcopy(layer) for _ in range(N)]).`

## TranformerEncoder

The `TransformerEncoder` glues everything together. Here, we define the `MultiHeadAttention`, `FeedForward`, `EncoderLayer`, and `Encoder` modules and build the complete forward pass. **Every line of code has already been presented during the previous sections**. All you have to do is copy-paste the correct snippets and glue everything together. 

Are you ready? Then dive right in.

> To test your implementation, you can construct a dummy input vector and a dummy mask and run a forward pass of your model.

Implement your own transformer in Pytorch.

Learn Deep Learning

Neural Networks

Training Neural Networks

Convolutional Neural Networks

Recurrent Neural Networks

Autoencoders

Generative Adversarial Networks

Attention and Transformers

Graph Neural Networks

Conclusion

Final Quiz

Build a Transformer Encoder

Linear layers

Layer normalization