Gain insights into basic and intermediate deep learning concepts, including CNNs, RNNs, GANs, and transformers. Delve into fundamental architectures to enhance your machine learning model training skills.

docker.tar.gz

Pytorch

GANS

This course is an accumulation of well-grounded knowledge and experience in deep learning. It provides you with the basic concepts you need in order to start working with and training various machine learning models. 

You will cover both basic and intermediate concepts including but not limited to: convolutional neural networks, recurrent neural networks, generative adversarial networks as well as transformers.

After completing this course, you will have a comprehensive understanding of the fundamental architectural components of deep learning. Whether you’re a data and computer scientist, computer and big data engineer, solution architect, or software engineer, you will benefit from this course.

Introduction to Deep Learning & Neural Networks

# How does LSTM work? 

LSTM (Long Short Term Memory) cells are the most commonly used RNN cell nowadays. Don’t be scared of the math! We will slowly clarify every term by inspecting every equation separately.

LSTM is the most popular RNN because it has a special memory component denoted by $c$ in the equations. It represents long-term memory.

Here is a sketch overview:

Let's start with some notations.

# Notation

> Before we begin, note that in all the equations, the weight matrices (W) are indexed, with the first index being the vector that they process, while the second index referring to the representation (i.e., input gate, forget gate).

To avoid confusion and maximize understanding, we will use the common notation: matrices are depicted with capital letters while vectors are represented by lowercase letters. For the element-wise matrix multiplication, the dot with the outer circle symbol $\odot$ is used.


- $c_t$ refers to the long term memory factor at timestep $t$, which is the main improvement of LSTM cells compared to plain RNNs.

- $x_t \in R^{N}$ is the input vector with $N$ elements at timestep $t$.
- ${h}_{t} \in R^{H}$ denotes the hidden RNN vector at timestep $t$. In the beginning, we initialize the vector with zeros for the first element of the sequence. $H$ is the hidden state dimension size.
- $\sigma$ is the sigmoid non-linear activation function.

# Equations of the LSTM cell

For $x_t \in R^{N}$ , where N is the feature length of each timestep, while $i_t,f_t,o_t,h_t,h_{t-1},c_t,c_{t-1},b  \in R^{H}$ , where H is the hidden state dimension, the LSTM equations are the following:

$$ i_t = \sigma( W_{xi} x_t + W_{hi} {h}_{t-1} + {W}_{ci} {c}_{t-1} + {b}_i)  \quad\quad(1)$$

$$ {f}_t = \sigma( {W}_{xf} {x}_t + {W}_{hf} {h}_{t-1} + {W}_{cf} {c}_{t-1} + {b}_f) \quad\quad(2) $$

$$ {c}_t = {f}_t \odot {c}_{t-1} + {i}_t \odot tanh( {W}_{xc} x_t + {W}_{hc} {h}_{t-1} + {b}_c ) \quad\quad(3) $$

$$ {o}_t = \sigma( {W}_{xo} {x}_t + {W}_{h0} {h}_{t-1} + {W}_{co} {c}_{t} + {b}_o)  \quad\quad(4) $$

$$ {h}_t = {o}_t \odot tanh({c}_t) \quad\quad(5)$$


The LSTM cell equations were written based on Pytorch documentation because you will probably use the existing layer in your projects.

Let's find out what these equations are actually doing.

# Equation 1: The input gate

$${i}_t = \sigma( {W}_{xi} {x}_t + {W}_{hi} {h}_{t-1} + {W}_{ci} {c}_{t-1} + {b}_i)  $$

Notice the following:
- The depicted weight matrices represent the memory of the cell.
- You see that the input ${x}_t$ is in the current input timestep, while $h$ and $c$ **are indexed with the previous timestep**.
- Every matrix W is a linear layer (we saw this in the first chapter ```nn.Linear(N,H)```). 

This equation enables us to the following:

1) Take multiple linear combinations of $x,h,c$.
2) Match the dimensionality of input $x$ (N) to the one of $h$ and $c$ (H). 

>The dimensionalities of $h$ and $c$ are the hidden states parameter in PyTorch's LSTM layer.

Moving on, the bias term is part of the linear layer and is simply a trainable vector that is added. The output is also in the dimensionality of the hidden and context vector (h).

Finally, after the 3 linear layers from different inputs, we have a non-linear activation function to introduce non-linearities, which enables the learning of more complex representations. 

In this case, the sigmoid function (```nn.Sigmoid()```) is usually used.

# Equation 2: The forget gate

$${f}_t = \sigma( {W}_{xf} {x}_t + {W}_{hf} {h}_{t-1} + {W}_{cf} {c}_{t-1} + {b}_f) $$

Simply, equation 2 is almost the same as equation 1. However, note that **the weight matrices are different this time**. This means that we get a different set of linear combinations that represent different things! The equations might be the same but  we want to model different things, as you will see.

# Equation 3: The new cell/context vector

$$ {c}_t = {f}_t \odot {c}_{t-1} + {i}_t \odot tanh( {W}_{xc} x_t + {W}_{hc} {h}_{t-1} + {b}_c )  $$

We have already learned a representation that corresponds to the info that we will “forget”. We also learned a representation $i$ for modeling the “input vector”. Let’s keep them aside and first inspect the $tanh$ parenthesis.

## The tanh() part  

Here, we have another linear combination of the input and hidden vector, which is again totally different! This term is the **new cell information**, which is passed by the $tanh$ function so as to introduce non-linearity and stabilize training. 

But we don’t want to simply update the cell with the new states. 

Intuitively, we want to take into account previous states; that’s why we designed RNNs in the first place!

> This is where the calculated input gate vector $i$ comes into play. We filter the new cell info by applying an element-wise multiplication with the input gate vector $i$.

## Injecting the forget gate information

The forget gate vector comes into play now. Instead of just adding the filtered input info, we first perform an element-wise vector multiplication with the **previous** context vector. 

To this end, we would like the model to mimic the forgetting notion of humans as a multiplication filter. 

By adding the previously described term in the $tanh$ parenthesis, we get the new cell state, as shown in equation 3. 

But what about the output of the LSTM cell in a single timestep?

# Equation 4: The almost new output

$$ {o}_t = \sigma( {W}_{xo} {x}_t + {W}_{h0} {h}_{t-1} + {W}_{co} {c}_{t} + {b}_o)   $$

It’s simple! Let’s just take another linear combination! This time, we combine our 3 vectors $x_t, h_(t-1), c_t$, while we add another non-linear sigmoid function in the end.

> Note that we will now use the calculated new cell state ($c_t$) as opposed to equations 1 and 2. 

We have almost calculated the desired output vector $o_t$ of a single cell in a particular timestep.

# Equation 5: The new context

$$ {h}_t = {o}_t  \cdot tanh({c}_t) $$

Note that the title in equation 4 was almost the new output.

Thus, one can calculate the new output (the new hidden state) based on equation 5. 

Suppose that we want to somehow mix the new context vector $c_t$ (after another activation!) with the calculated output vector $o_t$. This is exactly the point where we claim that LSTMs model **contextual information**. 

Instead of producing an output as shown in equation 4, we further inject the vector called context. Looking back at equations 1 and 2, one can observe that the previous context was involved. In this way, information based on previous timesteps is involved. 

> This notion of context (long-term memory) enabled the modeling of temporal correlations in long-term sequences.


Dive into the mathematics and intuition of  Long Short Term Memory Cells (LSTM).

Learn Deep Learning

Neural Networks

Training Neural Networks

Convolutional Neural Networks

Recurrent Neural Networks

Autoencoders

Generative Adversarial Networks

Attention and Transformers

Graph Neural Networks

Conclusion

Final Quiz

LSTM: Long Short Term Memory Cells

How does LSTM work?

Notation

Equations of the LSTM cell