Error Backpropagation with Matrix Multiplication

Vectorize the operations

Can we use matrix multiplication to simplify all the laborious calculation? It helped earlier when we were doing multiple calculations to feed the input signals forward.

To see if error backpropagation can be made more concise using matrix multiplication, let’s write out the steps using symbols. By the way, this is called vectorizing the process.

Being able to express a lot of calculations in matrix form makes it more concise for us to write down. It also allows computers to efficiently do all the work because they take advantage of the repetitive similarities in the calculations that need to be done.

The starting point is the errors that emerge from the neural network at the final output layer. Here, we only have two nodes in the output layer. These are $e_1$ and $e_2$:

$$\text{error}_\text{output} = \begin{bmatrix} e_1\\e_2 \end{bmatrix}$$

Next, we want to construct the matrix for the hidden layer errors. Let’s do it bit by bit. The first bit is the first node in the hidden layer. If we look at the diagrams above, we can see that the first hidden node’s error has two paths contributing to it from the output layer. Along these paths come the error signals $e_1 \cdot w_{11} / (w_{11} + w_{21})$ and $e_2 \cdot w_{12} / (w_{12} + w_{22})$. Now, look at the second hidden layer node, and we can again see two paths contributing to its error, $e_1 \cdot w_{21} / (w_{21} + w_{11})$ ...