Neural Network Building Blocks

Neural networks form a class of machine learning objects that implement a parameterized function composition. Tensors flow in a neural network by successive transformations called layers.

In this lesson, we’ll get familiarized with some of the most common types of layers:

• The linear layer

• The logistic, a.k.a the sigmoid function

• The ReLU function

• The leaky ReLU function

• The dropout layer

The linear layer

The linear layer is the star of neural networks. Almost every time we want to project a one-dimensional tensor, a.k.a. a vector, into another one-dimensional tensor, a linear layer will get involved.

A linear layer performs an affine transformation on a vector:

Here, $y$ is the output vector, $W$ is the weight vector, $b$ is the bias vector, and $x$ ...