Gain insights into basic and intermediate deep learning concepts, including CNNs, RNNs, GANs, and transformers. Delve into fundamental architectures to enhance your machine learning model training skills.

docker.tar.gz

Pytorch

GANS

This course is an accumulation of well-grounded knowledge and experience in deep learning. It provides you with the basic concepts you need in order to start working with and training various machine learning models. 

You will cover both basic and intermediate concepts including but not limited to: convolutional neural networks, recurrent neural networks, generative adversarial networks as well as transformers.

After completing this course, you will have a comprehensive understanding of the fundamental architectural components of deep learning. Whether you’re a data and computer scientist, computer and big data engineer, solution architect, or software engineer, you will benefit from this course.

Introduction to Deep Learning & Neural Networks

Mostly, the inability to fully understand transformers arises due to the confusion around secondary concepts. To prevent this from happening, we will gradually discuss all fundamental concepts and then construct a holistic view of transformers.  

With Recurrent Neural Networks (RNN’s), we used to treat sequences sequentially to keep the order of the sentence in place. To satisfy that design, each RNN component (layer) needs the previous (hidden) output. As such, stacked LSTM computations were performed sequentially. 

Then, transformers came out. 

The fundamental building block of a transformer is self-attention. To begin, we need to get rid of sequential processing, recurrency, and LSTMs.We can do that by simply changing the input representation.

## Representing the input sentence

### Sets and tokenization

The transformer revolution started with a simple question: 

> Why don’t we feed the entire input sequence so there are no dependencies between hidden states? That might be cool!

As an example the sentence “Hello I love you”:

This processing step is usually called **tokenization**, and it is the first out of three steps we need to perform before we feed the input in the model.


So instead of a sequence of elements, we now have a **set**. 

> Sets are a collection of distinct elements where the arrangement of the elements in the set does **not** matter. 

In other words, **the order is irrelevant**. We denote the input set as $X= [ x_1,  x_2,  x_3 … ,  x_N$] where ${X} \in R^{N \times d_{in}}$. The vectors (elements) of the sequence $x_i$ are referred to as **tokens**.

After tokenization, we project words in a **distributed geometrical space** or simply build word embeddings.


### Word embeddings
Words are not discrete symbols. They are strongly correlated with each other. This is why when we project them in a continuous space, we can find associations between them.

In general, an embedding is a representation of a symbol (word, character, sentence) in a distributed low-dimensional space of **continuous-valued** vectors. 

Dependent on the task, we can push word embeddings further away or keep them close together.

> Ideally, an embedding captures the semantics of the input by placing semantically similar inputs close together in the embedding space.

In natural language, we can find similar word meanings or even similar syntactic structures (i.e., objects get clustered together).


We know a priori that structure and order matter. To this end, we will devise a funky trick to provide some **notion of order** in the set.


### Positional encodings

When you convert a sequence into a set (**tokenization**), you lose the notion of order. 

Can you find the order of words (tokens) from the sequence “Hello I love you”? You most likely can.  But what about 30 unordered words? 

Remember, machine learning is all about scale. The neural network certainly cannot understand any order in a set. 

> Since transformers process sequences as sets, they are, in theory, **permutation invariant**.

Let’s help them have a sense of order by slightly altering the embeddings based on the position. Officially, **positional encoding** is a set of small constants that are added to the word embedding vector **before** the first self-attention layer. 

So if the same word appears in a different position, the actual representation will be slightly different **depending on where it appears in the input sentence**. 

Mostly, the inability to fully understand transformers arises due to the confusion around secondary concepts. To prevent this from happening, we will gradually discuss all fundamental concepts and then construct a holistic view of transformers.  

With Recurrent Neural Networks (RNN’s), we used to treat sequences sequentially to keep the order of the sentence in place. To satisfy that design, each RNN component (layer) needs the previous (hidden) output. As such, stacked LSTM computations were performed sequentially. 

Then, transformers came out. 

The fundamental building block of a transformer is self-attention. To begin, we need to get rid of sequential processing, recurrency, and LSTMs.We can do that by simply changing the input representation.

# Representing the input sentence

## Sets and tokenization

The transformer revolution started with a simple question: 

> Why don’t we feed the entire input sequence so there are no dependencies between hidden states? That might be cool!

As an example the sentence “Hello I love you”:

This processing step is usually called **tokenization**, and it is the first out of three steps we need to perform before we feed the input in the model.


So instead of a sequence of elements, we now have a **set**. 

> Sets are a collection of distinct elements where the arrangement of the elements in the set does **not** matter. 

In other words, **the order is irrelevant**. We denote the input set as $X= [ x_1,  x_2,  x_3 … ,  x_N$] where ${X} \in R^{N \times d_{in}}$. The vectors (elements) of the sequence $x_i$ are referred to as **tokens**.

After tokenization, we project words in a **distributed geometrical space** or simply build word embeddings.


## Word embeddings
Words are not discrete symbols. They are strongly correlated with each other. This is why when we project them in a continuous space, we can find associations between them.

In general, an embedding is a representation of a symbol (word, character, sentence) in a distributed low-dimensional space of **continuous-valued** vectors. 

Dependent on the task, we can push word embeddings further away or keep them close together.

> Ideally, an embedding captures the semantics of the input by placing semantically similar inputs close together in the embedding space.

In natural language, we can find similar word meanings or even similar syntactic structures (i.e., objects get clustered together).


We know a priori that structure and order matter. To this end, we will devise a funky trick to provide some **notion of order** in the set.


## Positional encodings

When you convert a sequence into a set (**tokenization**), you lose the notion of order. 

Can you find the order of words (tokens) from the sequence “Hello I love you”? You most likely can.  But what about 30 unordered words? 

Remember, machine learning is all about scale. The neural network certainly cannot understand any order in a set. 

> Since transformers process sequences as sets, they are, in theory, **permutation invariant**.

Let’s help them have a sense of order by slightly altering the embeddings based on the position. Officially, **positional encoding** is a set of small constants that are added to the word embedding vector **before** the first self-attention layer. 

So if the same word appears in a different position, the actual representation will be slightly different **depending on where it appears in the input sentence**. 

Represent text with positional encodings and embedding so it can be passed into a transformer. 

Key Concepts of Transformers

Represent text with positional encodings and embedding so it can be passed into a transformer.

Learn Deep Learning

Neural Networks

Training Neural Networks

Convolutional Neural Networks

Recurrent Neural Networks

Autoencoders

Generative Adversarial Networks

Attention and Transformers

Graph Neural Networks

Conclusion

Final Quiz

Key Concepts of Transformers

Representing the input sentence

Sets and tokenization