Computation Graphs

That automatic gradient calculation in the last section seems magical, but of course, it isn’t.

It is worth understanding just a little bit about how the automatic gradient calculation is done because this knowledge will help us later when we build larger networks.

Simple computation graph

Take a look at the following very simple network. It’s not a neural network, just a sequence of calculations.

In this illustration, we can see an input $x$, which is used to calculate $y$, which is then used to calculate $z$, the output.

Imagine that $y$ and $z$ are calculated as follows:

$$y = x^2$$

$$z = 2y + 3$$

If we want to know how the output $z$ varies as $x$ varies, we need to do some calculus to work out the gradient $\frac{dy}{dx}$. Let’s do this step by step.

$$\frac{dz}{dx} = \frac{dz}{dy}.\frac{dy}{dx}$$

$$\frac{dz}{dx} = 2.2x$$

$$\frac{dz}{dx} = 4x$$

📝 The first line is the chain rule of calculus, and it really helps us get started here. One of the appendices of Make Your Own Neural Network is an introduction to calculus and the chain rule if you need a refresher.

So we’ve just worked out that the output $z$ varies with $x$ as $4x$. If $x = 3.5$ ...