What is Autograd?

Overview

Autograd is an automatic differentiation package in the PyTorch library that helps train a neural network through graph computing. Instead of executing instructions immediately (also known as eager execution), Autograd builds a graph and uses it to speed up the calculation of derivatives, which are needed for training a neural network.

How it works

When training neural networks, weights and biases must be adjusted during backpropagation. This is done by finding the gradient of every output for every input. For an input vector x of n dimensions and an output vector of m dimensions, the matrix of these gradients would be:

This matrix, denoted by J, is called a Jacobian. Mathematically, Autograd calculates the Jacobianvector products.

For example, let's suppose we have another vector, v, and v is the gradient of the following scalar function:

$l$ = g(y): $v=\left(\begin{array}{ccc}\frac{\partial l}{\partial y_{1}} & \cdots & \frac{\partial l}{\partial y_{m}}\end{array}\right)^{T}$

Then, using the chain rule, the Jacobian-vector product gives a gradient l for x, as follows:

Recall that the chain rule is as follows:

Chain rule

In neural networks, the partial derivatives of the model’s outputs for its inputs would need multiple local partial derivatives for each multiplied learning weight, every activation function, and so on. This is computationally very expensive.

Autograd assists in solving this problem. It tracks all the operations performed on a tensor (vector) and stores them in the tensor itself. The tensor has a graph of its own that is made with the operations performed on it. This speeds up the process of derivative calculations.

Example

In the following code, we import the torch package for Autograd, and the matplotlib package for plotting graphs:

Next, we create a one-dimensional tensor with the flag requires_grad=True:

Here is the output:

Then, we perform the sin operation on the tensor and plot it. We use the .detach() method to stop tracking the operation so that it doesn't get included in the graph being built.

Here is the output:

As you can see, b is the sine of StartingVector. If we print b, we see that it has the history of its operations in it:

Here is the output:

The .grad_fn attribute contains information about the last operation. In this case, that operation is the sin operation.

Similarly, we can view the history of other operations:

Here is the output:

As seen above, c, d , and out have their operation information stored in grad_fn.

When computing derivatives, the loss function has a single value, so the out tensor also has only one value, 25.000, obtained by summing d.

We can view all past operations on d by using the grad_fn.next_functions method:

Here is the output:

To get the gradients, we use the .backwards() method:

Here is the output:

The graph gives us the value obtained by differentiating 2*sin(StartingVector) + 1, which is the operation performed on the input, from StartingVector.

Note: The gradients (.grad ) are only stored in leaf Nodes, that is, the input vectors. In this case, the input vector was StartingVector. Therefore, c.grad, d.grad, and so on will give None.

Enable or disable Autograd

When declaring Tensors for models using torch, requires_grad is assumed to be set toTrue. There are two ways of disabling this:

• Directly set the flag to False
• Use torch.no_grad

In such cases where Autograd is not enabled, the torch.enable_grad() method is used.

Pros and cons of using Autograd

Autograd runs code in graph execution mode as opposed to eager execution.

This has the following advantages:

• Since eager execution runs all operations one by one, it cannot take advantage of potential acceleration resources. Graph execution extracts tensor computations from Python and builds an efficient graph before evaluation.
• It allows for better parallel computing, since Autograd allocates resources more efficiently to run multiple operations in parallel. This also results in better utilization of GPUs or TPUs.

There are also some disadvantages to using Autograd:

• Autograd is unsuitable for smaller applications since it takes initial computing power to construct a graph.
• Depending on the implementation, the program can also become more complex.