Inside the Mind of a Neural Network

Mysterious black box

Neural networks are useful for solving the kinds of problems that we don’t know how to solve with simple rules. Imagine writing a set of rules to apply to images of handwritten numbers to decide what the number is. We can imagine that wouldn’t be easy, and our attempts would probably not be very successful either.

Once a neural network is trained and performs well enough on test data, we essentially have a mysterious black box.

This isn’t always a problem if we’re just interested in answers and don’t care how they’re arrived at. But it is a disadvantage of these machine learning methods—the learning doesn’t often translate into understanding or wisdom about the problem the black box has learned to solve.

Let’s see if we can take a peek inside our simple neural network to see if we can understand what it has learned and to visualize the knowledge it has gathered through training.

We could look at the weights, which is after all what the neural network learns. But that’s not likely to be that informative. Especially as the way neural networks work is to distribute their learning across different link weights. This gives them an advantage in that they are resilient to damage, just like biological brains are. It’s unlikely that removing one node, or even quite a few nodes, will completely damage the ability of a neural network to work well.

But, here’s an idea.

Backward query

Normally, we feed a trained neural network a question, and it outputs an answer. In our example, that question is an image of a human handwritten number. The answer is a label representing a number in the range $0–9$.

What if we turn this around and did it backwards? What if we fed a label into the output nodes, and fed the signal backwards through the already-trained network, until an image comes out from the input nodes? The following diagram shows the normal forward query, and this reversed backward query idea.