Accelerating AI

Handwritten Digit Recognition with FPGA-Powered CNNs

Discover how FPGA-powered Convolutional Neural Networks (CNNs) revolutionize handwritten digit recognition, achieving impressive speeds and accuracy. This article delves into the innovative implementation on the DE1-SoC platform, showcasing the seamless integration of hardware and advanced machine learning techniques.

Since Yann LeCun introduced the LeNet-5 model in 1998 for handwritten digit recognition, CNNs have been widely adopted. However, running these models in programming languages such as C or Python can be slow due to their sequential processing nature and memory bottlenecks. To address those issues, we implemented the CNN model for digit recognition on an FPGA with Verilog, a hardware description language that allows for hardware-level parallelism and efficient data management through on-board memory (SRAM). The implementation utilizes parallelization for both computations and memory accesses, enabling faster processing.

This article provides an overview of how we implemented a CNN model on the DE1-SoC, a hardware design platform that includes the Altera Cyclone V FPGA and dual-core Cortex-A9 embedded cores, utilizing hardware-level parallelism. It explains the CNN architecture, the DE1-SoC board we used, the communication between the FPGA and hard processor system (HPS), and outlines the digit recognition process.

Traditional Approach for Handwritten Digit Recognition

Earlier digit recognition algorithms include k-Nearest Neighbors (k-NN), Support Vector Machines (SVM), and basic Artificial Neural Networks (ANN). While k-NN is simple, it struggles with high-dimensional data and is computationally expensive. SVMs are effective for small datasets but require extensive tuning for larger, more complex data. Basic ANNs lack the capacity to automatically extract features, making them less accurate. In contrast, CNNs are designed specifically for image data. They automatically learn hierarchical features, which makes them significantly more accurate and efficient for tasks such as handwritten digit recognition.

In this article, we explore the representation of digit images in grayscale, where each pixel corresponds to an element in a matrix. The grayscale values are 8-bit, ranging from 0 (black) to 255 (white). Traditionally, recognizing digit images involves analyzing pixel-level features such as intensity, shape, and texture. A threshold is often applied to determine whether specific features exceed certain limits. This conventional approach operates without a loss function, which is used to guide the optimization process to improve accuracy and does not require labeled data, making it an unsupervised learning technique.

With the rapid advancements in machine learning, there is growing interest in applying these techniques to the problem of handwritten digit recognition, offering potential improvements over traditional methods. One of the most influential techniques is CNN.

CNN Overview

First, let’s have a look at how CNN works. It takes an image as input and produces a vector output representing the predicted class. By progressively transforming the input image through multiple layers, each specializing in different feature extraction tasks, CNNs can recognize complex patterns.

As shown in Figure 1, a good starting point to introduce CNN is to list out the functional components: convolution layer, Rectified Linear Unit (ReLu) module, max-pooling layer and fully connected layer.

The convolution layer can be thought of as a feature detector. It performs feature detection by applying a filter (a small matrix of weights) over each section of the image, calculating the weighted sum, and producing a new value that highlights specific patterns, such as edges or textures, in that area. By sliding this filter across the entire image, it helps the model understand the unique features that make up different objects or digits. In our case, there are three 3×3 pixel filters sliding across the 28×28 pixel image, returning three 26×26 pixel images. The selection of image and filter sizes is influenced by the limited memory capacity of our board, which we’ll explore further in the next section.

After the image is filtered, the ReLU module helps the model focus on the important parts by “turning off” negative values, essentially removing irrelevant data and making the positive information stand out for further analysis. After applying the ReLu function, the positive values remain and negative values have been set to zero.

Imagine zooming out from an image and only keeping the most important parts. That’s what the max-pooling layer does. It reduces the size of the data by keeping only the most significant information from each section, which makes the network faster and more efficient without losing too much detail. Mathematically speaking, the max-pooling layer reduces the spatial dimensions by selecting the maximum value from each subregion of the feature map produced by the convolutional layer.

In the final step, the fully connected layer takes everything the previous layers have learned and makes the decision. It connects all the dots (or pixels, in this case) and outputs the final result, like recognizing the digit in a handwritten number. It’s the part that delivers the answer based on all the processed data. The output matrix size from the max-pooling layer is 13×13×3, containing a total of 507 values after flattening. The fully connected layer can be mathematically represented as Equation 1:

where: “ƒc” is the output of the fully connected layer, the number with the highest index will be picked as the recognition result. “A” is the weight matrix of the fully connected layer and “mp” the output from the max-pooling layer. The final recognition result is determined by selecting the index of the highest value in the “ƒc” output, which corresponds to the predicted digit.

Introduction to DE1-SoC

As shown in Figure 2, the DE1-SoC Development Kit is a hardware platform built around the Altera System-On-Chip (SoC) FPGA, which integrates two key components: the Hard Processor System (HPS) and the FPGA. The HPS contains two ARM Cortex-A9, handling general-purpose tasks, while the FPGA is used for custom hardware processing, allowing for efficient, high-speed computations. Together, these components provide a flexible and powerful design environment for tasks such as real-time image recognition and other advanced processing applications.

The HPS communicates with the FPGA by means of shared memory. Data transfer from the HPS to the FPGA is handled through memory mapping. On the FPGA side, communication happens via the AXI bus, which is an interface protocol that enables efficient communication between different components in a SoC. By defining the addresses for parallel ports (referred to as “pio ports”) and ensuring both sides are properly synchronized, we enable seamless data exchange between the HPS and FPGA. As shown in Figure 3, this setup facilitates smooth interaction between the two components in our design.

Training Our CNN Model with PyTorch

Before we could run our handwritten digit recognition system on hardware, we first had to test its performance using software and obtain the necessary weights for each layer of the system. We used PyTorch to generate and test our CNN weights. Instead of manually coding all the algorithms, we just needed to specify some attributes for CNN in PyTorch, and it automatically built the neural network for us. We trained our model using a portion of the MNIST dataset (a popular collection of handwritten digits) and tested it with another portion, achieving 99% accuracy. Afterward, we extracted the weights for both the convolutional and fully connected layers, which will be used when deploying the system on hardware. PyTorch also made it simple to gather these weights.

System Design

The C program running on the HPS manages data transfer between our computer and the handwriting recognition system. For input, we used images of digits drawn in Microsoft Paint.

Once the image is preprocessed by our C program, it gets sent to the FPGA. At the same time, the weights of the convolutional layer are also sent to the FPGA. The system starts when we press the built-in SoC button to issue a start signal to the FPGA. The FPGA then processes the image through the network. However, due to memory constraints—only three memory blocks are allocated—the FPGA halts processing just before the fully connected layer and sends a signal back to the HPS. This pause allows for memory swapping, where weights from the convolutional layer are replaced with the one from the fully connected layer to make room for subsequent computations.

After receiving the needed data, the FPGA completes the process and gives us 10 numbers. The index of the number with the highest value is the result. While we could apply a “softmax” function to calculate probabilities for each number, it’s not necessary during recognition—it’s mainly useful when training the system.

CNN Design in FPGA

In our convolutional layer design, we use one multiplier for each convolutional unit, which is the minimum component to conduct convolutional arithmetic. The mathematical expression for the convolution output element at position (i,j) is given by:

where: y_i,j is the output value at position (i,j) in the output feature map. x represents the input feature map. w represents the kernel (filter) weights. b is the bias term (optional, depending on the architecture). M and N are the height and width of the kernel. The sums are performed over all elements within the kernel window.

As illustrated in Figure 4, at least nine clock cycles are needed to perform the convolutional arithmetic. It starts with the top left pixel in the kernel. In the next cycle, move to the right to the next pixel for multiplication, and then accumulate with the previous value. This process is repeated until the entire convolutional unit is calculated, and then outputs the accumulated value. Considering the state of idle and storing the data, we need 14 clock cycles to finish a convolutional calculation since each convolutional result should be put into the SRAM at a time.

In the convolutional layer, the arithmetic operations begin at the top-left corner of the image. The convolutional kernel then moves across the image, processing each 3×3 section one at a time. This process is repeated until the entire image has been covered, ending at the bottom-right corner. Once complete, the outputs from the convolutional layer are stored in the designated SRAM. To speed up the process, we implemented three parallel channels for concurrent processing. The control flow within the convolutional layers is managed by a state machine illustrated in Figure 5, ensuring seamless operation. In the IDLE state, image data and weights for both the convolutional and fully connected layers are loaded. The CALC state handles the computation for each convolutional unit, while the STORE state saves the computed output to SRAM and shifts the kernel to the next computation unit. Finally, the DONE state signals the max-pooling layer to begin its processing.

In our design of the max-pooling layer module, we imitated the convolutional layer control flow. The difference is that the internal state machine is quite simple, because the maximization operation can be done without complex logic. In the IDLE state, it waits for the start signal, which is also the end signal of the convolutional layer. It transits to the CALC state when it receives that signal. Next, it traverses the input matrix to find the maximum value in each 2×2 subregions. After that, the state transits into WRITE state and the maximum value will be stored into the appropriate destination SRAM address. Finally, it reaches the DONE state and gets ready for the next subregion processing.

The fully connected layer converts the 507 outputs from the max pooling layer into 10 output values. The index, which contains the highest output value, is the predicted number. However, As the weight matrix has a size of 507×507, it is not possible to upload the matrix to SRAM at one time because of limited memory. Our solution is to upload the matrix and calculate the output two times, as illustrated in the state machine in Figure 6. The final predicted number of the CNN will be shown in the digital display.

Results

Some of the results are shown here. The leftmost LED represents the output.

In Figure 7 and Figure 8, we show our system in action, successfully recognizing the handwritten digits 3 and 7. To put our design to the test, we used Microsoft Paint to draw 10 different images of each digit (0-9) and ran them through our recognition system. The results are summarized in Table 1, where you can see the accuracy for each digit.

Number	FPGA Accuracy	PyTorch Accuracy
1	90%	100%
2	100%	100%
3	90%	100%
4	100%	100%
5	70%	90%
6	80%	100%
7	90%	100%
8	100%	100%
9	90%	100%
0	100%	100%
Average	91%	99%

Overall, our system’s average accuracy is a bit lower than what we get from PyTorch. One reason for this difference could be the numerical formats: PyTorch uses single-precision floating-point numbers, which are more precise, while our FPGA system operates on 16-bit fixed-point numbers. This difference in precision might cause some slight variations in performance. One approach to resolving this issue is to maintain consistency in the numerical format between PyTorch and the FPGA, which is very likely to enhance accuracy. We might also use Quantization Aware Training provided by PyTorch, enabling us to simulate fixed-point effects during training and convert the trained model to a quantized fixed-point representation on Verilog.

Conclusion

We’ve documented the results of our project in a detailed video [2]. Our innovative design gives the CNN recognition algorithm a serious speed boost, with our FPGA-powered setup reaching an impressive 91% accuracy. Of course, there’s always room for further improvement—such as hooking up a real-time camera, switching from fixed-point to floating-point math, or training a fancier model in PyTorch for better weight matrices and performance.

If we’re feeling adventurous, we could complicate things even further by throwing in a few extra layers between the input and output. But hey, why make things simple when you can make them complex, right? 

REFERENCES
[1] Cyclone V Hard Processor System Technical Reference Manual https://www.intel.com/content/www/us/en/docs/programmable/683126/21-2/introduction-to-the-hard-processor-system-98309.html

[2] The Videos linked here show the result of our project:

RESOURCES
Altera | www.altera.com
Intel | www.intel.com

Code and Supporting Files

PUBLISHED IN CIRCUIT CELLAR MAGAZINE • MARCH 2025 #416 – Get a PDF of the issue

Sponsor this Article