The Rise of Deep Learning

In the 1990s, neural network research was in a rut again. After the discovery of backpropagation, connectionists had a few high-profile successes, including a system to read handwritten numbers for the U.S. Postal Service. And yet, the AI community was still skeptical of neural networks.

The general consensus was that the regular neural networks could solve simple problems—but that was about as far as they could go. To tackle more interesting problems, we needed deep neural networks, and those were very complicatedThey were slow to train, prone to overfitting and riddled by frustrating problems such as the vanishing gradient. Connectionists had cracked backpropagation, but they still could not win over their peers.

Then a few things happened that changed everything.

Building up to a perfect storm

The progress of machine learning from the 1990s to the early 2010s is a very fascinating story. In spite of the general skepticism around neural networks, a covenant of researchers kept advancing the field. Today, some of the famous names in this field are Geoffrey Hinton, Yahn LeCun, Yoshua Bengio, and many others.

One problem after the other, those stubborn pioneers tackled the most pressing issues of deep networks. They discovered novel weight initialization methods and activation functions like ReLUs to counter vanishing and exploding gradients. They sped up networks with better optimization algorithms and better activation functions. They ...