From the Chain Rule to Backpropagation

Backpropagation is an application of the chain rule, it is one of the fundamental rules of calculus. The chain rule calculates the gradient of any node y with respect to any other node $x$, we multiply the local gradient of all the nodes on the way back from $y$ to $x$.

Let’s understand how the chain rule works on a couple of network-like structures: a simple one, and a more complicated one.

The chain rule on a simple network

Let’s look at this simple network-like structure:

This is not a neural network, because it does not have weights. Let’s borrow a term from computer science, and call it a computational graph. This graph has an input $a$, followed by two operations: multiply by two and square. The output of the multiplication is called $b$, and the output of the entire graph is called $c$.

Now let’s say that we want to calculate ∂$c$/∂$a$, the gradient of $c$ with respect to $a$. Intuitively, that gradient represents the impact of $a$ on $c$. Whenever $a$ changes, $c$ also changes, and the gradient measures the amount of change. (If you find the gradientcomplex, review the Gradient Descent lesson).

For such a small graph, we could calculate ∂$c$/∂$a$ in a single step, by taking the derivative of $c$ with respect to $a$. However, as we mentioned earlier, that derivation would become impractical for very large graphs. Instead, let’s calculate the gradient using the chain rule, which works for graphs of any size.

Here is how the chain rule works. To calculate ∂$c$/∂$a$:

1. Walk the graph back from $c$ to $a$.
2. For each operation along the way, calculate its local gradient—the derivative of the operation’s output with respect to its input.
3. Multiply all the local gradients together.

Let’s see how that process works in practice. In our case, the path back from $c$ to $a$ involves two operations:

• A square
• A multiplication by $2$.

Let’s compile the local gradients of those two operations:

How do we know that ∂$b$/∂$a$ is $2$, and ∂$c$/∂$b$ is $4a$? Well, even though we use the chain rule, we must still compute the local gradients in an old-fashioned way, by taking derivatives by hand. However, don’t worry if you do not know how to take derivatives. We can always use libraries to do that. For now, we just have to understand the process.

Now that we have the local gradients, we can multiply them to get ∂$c$/∂$a$:

$$\large{ \frac{\partial c}{\partial a} = \frac{\partial c}{\partial b}.\frac{\partial b}{\partial a} = 4a.2 = 8a}$$ ...