The Development Cycle of Neural Networks

The development (testing) cycle

To see where the testing hurdle is, we’ll tune our neural network with an iterative process. We‘ll perform the following steps:

1. Tune the network’s hyperparameters.
2. Train the network on the training set.
3. Test the network on the test set.
4. Repeat until we are happy with the network’s accuracy.

This process is similar to the ML equivalent of software development, so we can simply call it the development cycle.

We already went through a few iterations of development in the previous chapter, when we measured the network’s performance with different batch sizes. However, we overlooked a distressing fact that the development cycle violates the blind test rule. Here’s why:

During development, we tune the neural network’s hyperparameters while looking at the network’s accuracy on the test set. By doing that, we implicitly custom-tailor the hyperparameters to get a good result on that set. In the end, our hyperparameters are optimized for the test set and are unlikely to be equally good on never-before-seen ...