LeNet-5 — A complete guide

In 1998, Yann LeCun and team introduced LeNet-5, a groundbreaking convolutional neural network (CNN) developed at AT&T Labs. Its mission was to revolutionize handwritten character recognition, particularly in banking, where processing checks was a labor-intensive task. Thanks to LeNet-5, the accuracy of reading handwriting went way up, and the number of mistakes went way down. This big change had a huge impact on how banks worked, and it changed them forever. LeNet-5’s versatility extended across industries, automating tasks from address recognition to document digitization. Let’s look at its architecture and explore how this compact neural network reshaped the landscape of AI.

Understanding the evolution of neural networks, particularly milestones like LeNet-5, is essential for machine learning engineers and data scientists as it provides insight into the foundational principles of modern deep learning architectures. This historical context not only enriches one’s understanding of neural networks but also equips professionals with the knowledge to innovate and adapt these principles in their own work, fostering career advancement in the rapidly evolving field of machine learning.

LeNet architecture#

LeNet is characterized by a simple yet innovative architecture featuring a series of layers that collectively enable effective feature extraction and hierarchical learning.

The key components of the basic LeNet structure include:

1. Input layer: LeNet typically takes grayscale images of fixed dimensions as input. In the case of the original LeNet-5, the input size is $32$ x $32$ pixels.

2. Convolutional layers (C₁ and C₃): LeNet employs convolutional layers with learnable filters that automatically extract features from the input image. The convolutional operation involves sliding these filters across the input, capturing spatial hierarchies and patterns.

3. Subsampling layers (S₂ and S₄): After convolution, LeNet introduces subsampling layers to downsample the spatial dimensions of the feature maps. This reduction aids in building translation invariance, making the network robust to slight shifts in the input.

4. Fully connected layers (F₅ and F₆ ): In this layer, the extracted features are flattened and fed into dense layers for classification.

5. Output layer: Using the softmax activation functionThe softmax activation function converts raw numbers into probabilities, ensuring they add up to 1. It's used in the output layer of neural networks for classification tasks to interpret predictions as probabilities., the output layer produces probabilities for each class.

LeNet’s impact on the development of modern CNNs#

LeNet’s influence on modern CNNs cannot be overstated. Its success in handwritten digit recognition showcased the potential of convolutional neural networks in practical applications. LeNet introduced fundamental concepts, such as convolutional and subsampling layers (pooling layers), that proved crucial for learning hierarchical representations of features.

The principles established by LeNet served as the foundation for subsequent advancements in deep learning and CNN architectures. Many modern convolutional neural networks, including popular ones like AlexNet, VGGNet, Inception (GoogLeNet), ResNet, EfficientNet, MobileNet, and YOLO, have drawn inspiration from LeNet’s principles, evolving and optimizing them for various image recognition tasks.

LeNet-5 paved the way for a new era in computer vision, influencing the design and development of neural networks that power state-of-the-art solutions in image processing, object detection, and beyond. Its legacy continues to resonate in the ongoing exploration and refinement of deep learning models.

Layers in LeNet architecture#

The architecture of LeNet comprises multiple layers, each playing a crucial role in the hierarchical feature extraction process. Let’s dive into the details of these layers:

1. Convolutional layer (C1)#

The convolutional layer (C₁) serves as the initial building block in the LeNet architecture, playing a crucial role in extracting features from the input image.

In the convolutional operation, C₁ utilizes learnable filters or kernels to scan the input image. These filters slide across the input, performing element-wise multiplications and aggregating the results. The output of this operation, known as a feature map, captures local patterns and features from the input. The convolutional operation is pivotal for learning hierarchical representations of visual information, allowing the network to recognize complex patterns.

Activation function and pooling layer#

Following the convolutional operation, an activation function is applied to introduce non-linearity into the network. In LeNet, the activation functions used in the early layers are typically sigmoidThe sigmoid activation function squeezes any input value to a range between 0 and 1, making it useful for binary classification tasks in neural networks. or tanhThe tanh (hyperbolic tangent) activation function squashes input values to a range between -1 and 1, making it useful for neural networks dealing with data that spans negative and positive values.. These functions squash the output of the convolutional operation into a bounded range, enabling the network to model complex relationships and capture non-linear patterns in the data.

After the convolutional operation, LeNet incorporates subsampling layers, often referred to as pooling layers. The purpose of subsampling is to downsample the spatial dimensions of the feature maps, reducing computational complexity and enhancing the network’s translational invariance. Max pooling is a commonly used technique in LeNet, where the maximum value in each local region is retained, effectively highlighting the most significant features while discarding less relevant information. The pooling layers contribute to the network’s ability to recognize patterns regardless of their precise spatial location, making LeNet more robust to variations in the input data.

2. Convolutional layer (C3)#

The convolutional layer (C₃) in LeNet represents the second stage of feature extraction, building upon the foundation laid by the initial convolutional layer (C₁).

The second convolutional layer (C₃) serves as the successor to C₁, extending the hierarchical learning process by capturing more complex and abstract features from the previously extracted information. The introduction of a second convolutional layer allows the network to discern higher-level patterns and relationships within the input data.

Activation function and pooling layer#

Similar to the first convolutional layer, C₃ utilizes an activation function to introduce non-linearity into the network. Additionally, pooling is applied in C₃ to downsample (pooling layer) the spatial dimensions of the feature maps.

Explanation of feature maps and filter sizes#

Feature maps in the context of C₃ represent the output of the convolutional operation applied to the feature maps from the previous layer (typically the output of C₁ or a similar layer). These feature maps capture complex combinations of features learned by the network and serve as input for subsequent layers.

The filter sizes in C₃ determine the receptive field of the convolutional operation. Larger filter sizes allow the network to capture more global features, while smaller filter sizes focus on local patterns. The choice of filter sizes in C₃ influences the network’s ability to recognize and generalize patterns in the input data.

3. Fully connected layer (F5)#

The fully connected layer (F₅) marks a crucial transition in the LeNet architecture, representing the point where the hierarchical features extracted by the convolutional layers are flattened and fed into densely connected nodes.

After the convolutional layers (C₁ and C₃) have captured hierarchical features from the input, the fully connected layers introduce a transition to a more traditional neural network architecture. This transition enables the network to leverage the learned features for classification tasks.

Flattening of feature maps from previous layers#

Before entering the fully connected layers, the feature maps obtained from the convolutional layers are flattened. This process involves reshaping the multi-dimensional arrays into a one-dimensional vector. The flattened representation preserves the learned features' spatial hierarchy while transforming them into a format compatible with traditional neural network layers.

4. Fully connected layer (F6)#

The fully connected layer (F₆) in LeNet represents another key component in the network’s architecture, following the initial transition from convolutional layers to fully connected layers.

Another fully connected layer in LeNet#

F₆ continues the process of leveraging the hierarchical features extracted by the earlier layers for the final stages of classification. As a fully connected layer, F₆ is densely connected to the nodes from the preceding layer, allowing it to capture complex relationships and dependencies in the learned features.

Activation function and output size#

Similar to the previous fully connected layer (F₅), F₆ applies an activation function to introduce non-linearity into the network. The choice of activation function, commonly tanh or sigmoid, is critical for regulating the information flow through the network and facilitating the learning process.

The output size of F₆ depends on the number of nodes in this layer, which is typically determined based on the desired number of classes for the classification task. For example, in the case of handwritten digit recognition using the MNIST dataset, F₆ might have $10$ nodes, each corresponding to one digit ($0$ through $9$). The activation values of these nodes represent the network’s confidence or probability for each class.

The activation function in F₆ helps in squashing the output values to a specific range, allowing the network to produce meaningful probabilities for each class. This probability distribution is then used to make predictions and classify the input data.

5. Output layer#

The output layer in LeNet is the final component of the network, responsible for producing the classification results based on the features processed through the preceding layers.

As the terminal stage of the LeNet architecture, the output layer takes the features extracted and processed by the preceding layers and translates them into a meaningful prediction. This layer is specifically designed for the classification task, providing probabilities or scores for each possible class.

Activation function#

The activation function employed in the output layer is typically the softmax function. Softmax is well-suited for multiclass classification problems, as it transforms the raw output scores into a probability distribution. This distribution ensures that the sum of probabilities for all classes is equal to one, allowing for a clear and interpretable output.

The softmax activation function takes the raw output values from the previous layer and normalizes them, converting them into probabilities. This normalization enables the network to express its confidence in each class, making it easier to interpret the final output.

Output interpretation and use of class labels#

The output from the output layer represents the network’s prediction for the input data. Each node in this layer corresponds to a specific class, and the associated probability indicates the network’s confidence in assigning the input to that class.

In the context of classification tasks, the class label with the highest probability is chosen as the predicted class. The use of class labels facilitates the interpretation of the model's predictions and enables users to understand which specific category the input data belongs to.

For instance, in the case of LeNet applied to the MNIST dataset for handwritten digit recognition, the Output Layer would have 10 nodes, each representing a digit from 0 to 9. The class label associated with the node having the highest probability would be the predicted digit.

Step-by-step implementation of LeNet-5#

Let’s dive into the detailed implementation of LeNet-5 for MNIST digit classification. By following each step carefully, we’ll gain a comprehensive understanding of how to build and train this powerful convolutional neural network. This walkthrough will equip us with practical insights into leveraging LeNet-5 for accurate digit recognition tasks, paving the way for broader applications in image processing and pattern recognition.

Importing libraries#

We start by importing necessary libraries such as NumPy, Matplotlib for visualization, scikit-learn for metrics, and TensorFlow for building and training the neural network model.

Loading and preprocessing the MNIST dataset#

We load the MNIST dataset using TensorFlow’s built-in datasets module. The dataset is split into training, validation, and testing sets. We preprocess the data by reshaping it into the required format and normalizing pixel values to fall within the range $[0, 1]$.

Building the LeNet-5 model#

The LeNet-5 model architecture is constructed using TensorFlow’s Keras API. It consists of convolutional layers with ReLU activationReLU (Rectified Linear Unit) activation function outputs the input value if it's positive, otherwise, it returns zero., max-pooling layers, and fully connected layers. The model is compiled with appropriate loss function, optimizer, and evaluation metric.

The following is the LeNet-5 architecture built using the above code:

Training the model#

The constructed model is trained using the training data with a portion reserved for validation. The training process is visualized by plotting training and validation accuracy and loss over epochs.

The following is the output of the aforementioned code:

Testing the model#

The trained model is evaluated on the test dataset to assess its performance on unseen data. Test accuracy and loss are calculated, and a classification report is generated to analyze the model’s performance in detail.

The following is the output of the aforementioned code:

Conclusion#

LeNet-5 revolutionized convolutional neural networks, particularly in handwritten digit recognition. Its hierarchical feature learning and efficient architecture paved the way for modern CNNs. Through step-by-step implementation, we showcased its practicality in MNIST digit classification, achieving high accuracy. LeNet-5’s legacy persists, shaping the landscape of deep learning and driving innovation in artificial intelligence.

Next steps#

If you want to expand your knowledge and learn more about CNNs, the following courses are an excellent starting point for you: