The Case for Backpropagation

Recap and follow up of our neural network

Before we jump into the backpropagation, let’s have a quick review of what we have learned about neural networks so far.

In the previous chapter, we wrote a functioning neural network, or at least half of it. The network’s prediction code is done: it passes data through the model, and produces labels. However, that process also requires a set of weights, and we haven’t written the code that finds those weights yet. We’ll do it in this chapter by implementing the train() function of our neural network.

In the early years of neural networks, training was tough. AI experts even questioned whether they could be trained at all. The answer came in the early 1970s, when researchers found a way to calculate the gradient of a network with an algorithm called backpropagation or backprop.

There is a chance that we’ll never have to implement backpropagation in a real-life project. Modern ML libraries already ...