Using an ESP32 MCU and Apache HTTP Server
In this article, I describe a firmware update over-the-air (FUOTA) process, using the ESP32 system-on-a-chip (SoC) with Wi-Fi connectivity. I also explain how to establish a simple Web server to host firmware binaries and send them to the microcontrollers (MCUs).
Ideally, embedded software, or “firmware,” is as reliable as possible. Being bug-proof, immune to security failures, and efficient are highly desirable features in embedded software development. But real life is tough for embedded software. It’s impossible to prevent and fix every possible malfunction the software could potentially experience during operation. No matter how well-designed the software is, there is always the potential for failure due to unforeseen bugs.
Fortunately, for embedded systems with Internet connectivity (Ethernet or Wi-Fi), there’s an ultimate solution for this: a firmware update over-the-air (FUOTA). FUOTAs update embedded systems’ firmware remotely, so that a bug or failure can be fixed quickly in the field without any costs related to human intervention. All it requires is an Internet connection and the capability to download and flash new firmware.
This article will show how to update firmware remotely in ESP32 SoC-based embedded systems.
FIRST OF ALL, WHAT IS AN ESP32?
ESP32 is a family of low-cost, SoC (system-on-a-chip) microcontrollers (MCUs) with wireless connectivity (Wi-Fi and Bluetooth), manufactured by Espressif Systems. Several models and variations are available, providing suitable options for driving embedded systems that need wireless connectivity out of the box.
Regarding hardware, ESP32 SoCs are available as both single- and dual-core CPUs (usually with 240MHz clock frequency), with significant RAM size (usually 512KB) and flash memory ranging from 4MB to 16MB SPI-flash. The ESP32 is therefore a good choice for mid- and high-complexity embedded systems.
From the standpoint of software development kits (SDK), the ESP32 APIs, examples, and repositories are well documented and have solid support from the Espressif Systems and developer community. This SDK is called ESP-IDF (IDF stands for IoT Development Framework), and works natively using FreeRTOS as the embedded real-time operating system [1].
— ADVERTISMENT—
—Advertise Here—
HOW A FUOTA WORKS: PROCESS OVERVIEW
The capability of an MCU and/or SoC to update its own firmware over the Internet can be a life saver when electronics devices face bugs, malfunctions, and security failures during operation. But how does such an update work? Let’s start with an overview of the FUOTA process.
First, based on one or more specific stimuli (such as a command, a standard verification after boot, or an event), the MCU/SoC makes a request over the Internet to a remote server for new firmware. Usually, this request contains some sort of firmware and security control, such as the current firmware version flashed on the MCU, a security token, or a firmware hash.
Second, the remote server validates this request. If this validation goes well, it sends back to the MCU/SoC a new firmware file (also referred to as a firmware binary).
Third, the MCU/SoC downloads the new firmware and stores it, usually on non-volatile memory such as flash memory.
Fourth, after the firmware download completes, the MCU/SoC can check firmware integrity and apply security verifications. If everything’s all right, the MCU/SoC flashes itself with this new firmware. This step can include a swap operation, which means flashing the new firmware over the flash memory session or partition used for the current firmware.
Finally, after flashing the new firmware, the MCU reboots, and the new firmware is in operation.
As can be seen in this process, updating firmware relies on both MCU capabilities and remote server availability. This means it’s crucial for a FUOTA that both the MCU and remote server are available 24/7. Therefore, the MCU/SoC firmware update process should be extensively tested, to ensure the availability of the remote server.
HOW A FUOTA WORKS: INTERNET CONNECTIVITY
Now, let’s expand on the key feature of updating the firmware over the air: Internet connectivity. First of all, the MCU/SoC must be able to handle Internet connectivity, which implies a stable TCP/IP stack or support should be used. Fortunately, MCUs and SoCs today can count on good, open-source TCP/IP stacks, such as lightweight IP (lwIP).
Regarding MCU storage and flash firmware operations, it’s strongly recommended to partition flash memory. A partitioned flash memory consists of splitting the whole flash memory into subsections called partitions, where each partition is responsible for storing data for a well-defined purpose. It works much like partitioned hard drives on PCs. The basic partition scheme used for flash memory partitions is shown in Figure 1. The flash memory has three partitions—bootloader, App (Area 1), and OTA (Area 2).
— ADVERTISMENT—
—Advertise Here—
Basic partition scheme for a FUOTA.
The bootloader partition stores a specific piece of software called “bootloader.” It is responsible for booting the MCU, setting up its peripherals, and carrying out manufacturer-specific operations. Some MCUs work with two bootloader stages—the first stage is not mutable (written on the MCU by the manufacturer), while the second stage can use a bootloader implemented by developers. Bootloader software is also responsible for pointing the MCU’s CPU to the first main firmware address, in order for the CPU to be able to start executing flashed firmware.
The App partition is where the main firmware is stored. This partition must be large enough for storing the firmware and (maybe) some persistent data, such as configurations and parameters required for the firmware to work. Considering future expansions and new implementations over the embedded software life cycle, it’s good practice to use a flash memory that allows for a larger-than-required App partition. This prevents hardware from becoming obsolete due to insufficient flash memory space.
The OTA partition stores the firmware downloaded from the remote server, and therefore it must be the same size as the App partition. If it’s larger than the App partition, flash memory storage will surely be wasted; if it’s smaller, there’s a strong chance the downloaded firmware won’t fit on it, resulting in a failure to update.
Given the OTA and App partitions, there are two ways to handle the firmware update. The first and most intuitive option is to use the OTA partition as a temporary place to store the firmware when downloading it. Then, after download completes, its content is simply copied over to the App partition in what’s called a swap process.
The second (and smarter) option is to use the OTA and App partitions in firmware-fall-back functionality, known as A/B system updates. In A/B system updates, the OTA and App partitions are alternately used to store the main firmware each time an update is done, swapping roles with each subsequent firmware update. Let’s look at this option more closely.
Macro Steps for A/B System Updates
Before the first firmware update, the App partition contains the main firmware and the OTA partition is empty. After the first firmware update, the partitions’ functionalities change; The OTA partition is selected by bootloader as the partition for the main firmware, and the App partition will have its content, the previous firmware version, preserved. In the next FUOTA process, the partitions alternate: the App partition is selected by bootloader as the new partition for the main firmware, and the OTA partition will now have its content preserved.
After the second firmware update, partition functionalities change again. App is used (chosen by bootloader) as the partition containing the main firmware, while OTA contains the previous version of firmware, and is reserved to get the next firmware binary for the next firmware update process.
The A/B system updates provide a major advantage: the MCU can recover itself from an unsuccessful firmware update process. If a firmware update fails for some reason—for example, the firmware falls into a non-stop reboot cycle, or some critical failure happens every time the firmware runs—and boot cannot be completed, the device won’t be lost. The bootloader can point to the other partition (OTA or App) as the main firmware partition, to use the previous (and working) firmware version. This feature is particularly valuable for embedded devices operating in the field—especially high-hazard areas or areas that are difficult to access—when human intervention is not possible.
HOW FUOTA HAPPENS ON ESP32
The flash memory on an ESP32 can be partitioned in a similar way to that described in Figure 1. In fact, an ESP32’s partition scheme is flexible and can support many possible partitionings [2].
In terms of the software development environment for the project described in this article, the official APIs of Arduino IDE and Espressif’s ESP-IDF are used together to get the best of both worlds—Arduino IDE’s portability and ease of use, and the reliability of Espressif’s ESP-IDF. This approach will make the software development environment setup much easier to configure. Also, this works out of the box on any operating system. You can focus on the FUOTA ESP32 API usage and learn the fundamentals of FUOTA processes, instead of worrying about the tricky and not-so-trivial software environment setup that may be required in some cases when using “pure” ESP-IDF. However, after you get used to the FUOTA ESP32 API, I encourage you to learn how to reproduce everything shown here using ESP-IDF only, so as to use as much official SDK software APIs and libraries as possible.
ESP32 has excellent FUOTA APIs, which enable developers to make flexible and reliable wireless firmware updates. For instance, both the swap process and A/B system updates are supported. In this article, the swap process is used. Thus, firmware will be downloaded into the OTA partition, and after download successfully completes, the OTA partition’s content will be flashed over to the App partition, replacing the firmware stored there.
All the ESP-IDF APIs and functions used for the ESP32 FUOTA in this article can be found on Arduino-ESP32 official support by Espressif [3].
Also, it is important to know that in this article a firmware file/binary is sent to ESP32 from a Web server via HTTP, as a response to an HTTP GET sent by ESP32. The HTTP data traffic isn’t encrypted or protected, so all HTTP requests and responses can be sniffed using proper networking software tools. If you want to prevent your firmware binary from being sniffed or exposed to a malicious agent during a FUOTA process, you should check HTTP’s FUOTA process. This won’t be covered in this article, because it’s complex and requires all the content given here as a starting point. Therefore, this article is what you need to start making more secure FUOTA processes.
ESTABLISHING A WEB SERVER FOR HOSTING THE ESP32 FIRMWARE
So far, you have learned what a FUOTA is and how it works in an MCU. Now, it’s time to know a little more about the remote server side. Briefly, the remote server side consists of a Web server that hosts firmware binaries and sends them to MCUs/SoCs on demand. Also, the remote server can conduct additional checks to decide if the firmware request by an MCU will be accepted or declined. These include security checks, current and requested firmware versions checks, and password/token checks.
Each FUOTA remote server solution and its specific rules for the FUOTA process can be unique—there’s no standard at this point. In fact, it’s strongly recommended that the FUOTA process in real-world systems use sophisticated security checks, as attacks on the process can occur, such as flashing embedded systems with malicious firmware, or cloning a system’s firmware.
— ADVERTISMENT—
—Advertise Here—
If you aren’t familiar with state-of-the-art security checks, cryptography and other security measures that can be applied to the FUOTA process, you should start with the basics. A good first step is to establish a simple Web server to host firmware. This is the subject of this section.
The Web server used here is the Apache HTTP server, probably the most widely-used Web server in the world. To simplify the server-side setup, only the most basic elements of the Apache HTTP server will be used. However, keep in mind that the Apache HTTP server can go way beyond what’s shown here, and that it is also a good choice for robust and complex host solutions.
The good news for you is that every Linux machine with network access—even a Raspberry Pi single-board computer (SBC)— can support the Apache HTTP server. Further, most cloud host services, including inexpensive VPS services, can support it too. So the steps presented here to establish a Web server will work on both local networks or the Internet.
To establish a simple Web server to host ESP32 firmware files for FUOTA processes, follow these steps.
First, you must have a Linux machine with network access. As mentioned before, you can use something in your local network like a Raspberry Pi, or a cloud service such as a VPS service. For the Linux distribution (the operating system plus other components) in your machine, I’d like to recommend an Ubuntu “distro.” It’s stable and easy to use, especially if you’re not accustomed to working in Linux. The next steps of this procedure assume your Linux machine is using an Ubuntu distro.
Once you have a Linux machine, it’s time to install the Apache HTTP server. To do this, update your apt-get repository, and install the Apache HTTP server by executing the following commands:
sudo apt-get update
sudo apt-get install apache2This may take a while, depending on your Internet connection speed.
If you’re using a cloud hosting solution, you may need to adjust the Linux distro firewall to allow HTTP port (80) data traffic. Otherwise, your server may not be visible to the external world. To do this, list all the uncomplicated firewall (ufw) application profiles by executing the following command:
sudo ufw app listIf Apache is seen on this list, allow it access to port 80 by executing the following command:
sudo ufw allow ‘Apache’Finally, confirm the Apache application profile is allowed in the Linux firewall by executing this command:
sudo ufw statusYou should be able to see Apache on the list, followed by the action “ALLOWED” and “Anywhere” filled in the “From” field, which means the firewall now allows Apache to operate using port number 80.
Now your Apache HTTP server is ready for you to use. You can verify that Apache is working properly by accessing the server IP (or site address) in a browser. Once there, the browser page called “Apache2 Ubuntu Default Page” will show in your browser, meaning the Apache HTTP server is working.
To host any kind of file, including firmware binaries, in this Web server, you must place the it in the /var/www/html folder of your Linux machine. To test the download of this file, access the server IP (or site address) in a browser and add “/” plus the filename in the address. Once you hit “Enter,” your browser should start to download the file.
GETTING DEEPER INTO THE CODE: ESP32 FUOTA
Now, let’s get deeper into the code for the ESP32 FUOTA process by making an example project. This project is open source and available in a GitHub repository [4], which can be modified for your own needs. To reproduce this project, you’ll need an ESP32 DevKit V1, or any other ESP32 development kit with at least one GPIO to connect a push-button. You’ll also need a micro-USB cable, a push button, a protoboard, and some jumpers. Figure 2 shows the set up for this example project.
Set up of the example project
This project triggers a FUOTA process when it detects a push-button press. The ESP32 then requests a firmware binary to the remote server, the firmware is downloaded and flashed, and the ESP32 reboots with the new firmware.
The project uses a Wi-Fi network connection. The SSID and Password used for Wi-Fi authentication are defined in WIFI_SSID and WIFI_PASS, as shown in Listing 1.
The FUOTA remote server address and firmware filename are stored in host_http and firmware_filename variables, as shown in Listing 2. Note the format required for address and firmware filename.
The first actions this firmware performs are: initialize the serial interface, print the firmware version to the serial interface, initialize the FUOTA push button GPIO (input mode with internal pull-up resistor), and initialize and connect to the Wi-Fi network, as shown in Listing 3. In addition, the timestamp (the milliseconds since the ESP32 started) of the firmware init is taken in the firmware initialization.
Next, the loop() function executes the operations in Listing 4. The Wi-Fi connection state is constantly verified. When the ESP32 is disconnected from the Wi-Fi network, it’s automatically reconnected.
LISTING 1
The SSID and Password used for Wi-Fi authentication
#define WIFI_SSID “ “ /* Define here your wifi SSID */
#define WIFI_PASS “ “ /* Define here your wifi Password */LISTING 2
Storing the FUOTA remote server address and firmware filename
/* Insert here remote server address without http:// (example: www.example.com) */
String host_http = “”;
/* Insert here firmware filename considering its path in server (example: /firmware.bin) */
String firmware_filename = “”; LISTING 3
The first actions the firmware performs are: initialize the FUOTO push-button GPIO, initialize and connect to the Wi-Fi network, and store the timestamp that the firmware began to run.
/* Init serial (for verbose debugging) and writes current firmware version in it */
Serial.begin(115200);
Serial.print(“Firmware version: “);
Serial.println(FIRMWARE_VERSION);
/* Init FUOTA push-button GPIO */
pinMode(FUOTA_PUSH_BUTTON_GPIO, INPUT_PULLUP);
/* Init (empty) FUOTA URL */
url_FUOTA = “”;
/* Init and connect to wi-fi */
init_wifi();
/* Stores the timestamp that firmware started to run */
timestamp_since_firmware_start = millis();LISTING 4
The loop() function executes several operations.
void loop()
{
/* Ensure wi-fi connection is active */
verify_wifi_connection();
/* Check if FUOTA push-button has been pressed (considering debouncing).
If yes, FUOTA starts. */
FUOTA_must_start = false;
if (digitalRead(FUOTA_PUSH_BUTTON_GPIO) == LOW)
{
delay(FUOTA_PUSH_BUTTON_DEBOUNCE_TIME);
if (digitalRead(FUOTA_PUSH_BUTTON_GPIO) == LOW)
{
FUOTA_must_start = true;
}
}
if (FUOTA_must_start == true)
{
Serial.println(“FUOTA process - start”);
Serial.print(“Host: “);
Serial.println(host_http);
Serial.print(“Firmware filename: “);
Serial.println(firmware_filename);
init_FUOTA(host_http, firmware_filename);
}
/* Escreve periodicamente uma mensagem no serial monitor */
Serial.print(“Total time since firmware started: “);
Serial.print(calculate_time_difference(timestamp_since_firmware_start)/1000);
Serial.println(“s”);
delay(1000);The time spent since the firmware started is printed in the serial interface, so the firmware execution time can be checked.
The FUOTA push-button state is also constantly verified. When push-button pressing is detected and confirmed by the debouncing routine, init_FUOTA(), the FUOTA process function, is called, passing the remote server address (host_http) and firmware filename (firmware_filename) as parameters.
Now, let’s detail the init_FUOTA() function. This function is responsible for handling the entire FUOTA process: it requests the firmware to the remote server, downloads it, flashes it in the ESP32 flash memory, and reboots the ESP32 after the firmware flashing operation is done.
Requesting firmware to the remote server is done by sending a GET HTTP request to the remote server, receiving as response the firmware binary. The GET HTTP request process is shown in Listing 5.
Once an HTTP request response is received, it’s checked to ensure that it’s valid, as shown in Listing 6.
As seen in Listing 7, if the HTTP response is valid, the HTTP response data size, which corresponds to the firmware’s total size, is parsed. Also, the data type received in the HTTP request response must be application/octet-stream, which here corresponds to raw firmware data. This is also verified by the code shown in Listing 7.
If both firmware size and data type are valid, the FUOTA request response is valid and the FUOTA process starts. This is done by calling the Update.begin() function, an ESP32 API for preparing the FUOTA process:
FUOTA_can_start = Update.begin(FUOTA_firmware_size);
If Update.begin() returns true, it means the ESP32 is ready for a FUOTA. Then, the Update.writeStream() function, an ESP32 API function, is called, and passes as a single parameter the HTTP stream. The Update.writeStream() function downloads the firmware from the remote server and flashes it to the ESP32 flash memory:
FUOTA_can_start = Update.begin(FUOTA_firmware_size);
Update.writeStream() returns the bytes written to flash in the FUOTA process. If this number is equal to the HTTP response data size, the firmware size received in HTTP request response, it means the firmware is totally flashed to the ESP32 flash memory. Finally, Update.end(), also an ESP32 API function, checks if the FUOTA is finished. In a positive case, the FUOTA process was successful, and the ESP32 will reboot with its new firmware. This is shown in Listing 8.
LISTING 5
The GET HTTP process
espClient.print(String(“GET “) + firware_filename + “ HTTP/1.1\r\n” +
“Host: “ + host_http + “\r\n” +
“Cache-Control: no-cache\r\n” +
“Connection: close\r\n\r\n”);
/* Wait for FUOTA remote server response. Timeout is defined in FUOTA_SERVER_TIMEOUT */
timeout_fw_FUOTA = millis();
while (espClient.available() == 0)
{
if ( calculate_time_difference(timeout_fw_FUOTA) > FUOTA_SERVER_TIMEOUT )
{
Serial.println(“[ERROR] Timeout: no response received from FUOTA remote server. FUOTA is cancelled”);
espClient.stop();
Serial.println(“Rebooting ESP32 due to FUOTA error”);
ESP.restart();
}
}LISTING 6
Checking to ensure that that an HTTP request response is valid
if (http_line.startsWith(“HTTP/1.1”))
{
if (http_line.indexOf(“200”) < 0)
{
Serial.println(“[ERROR] FUOTA remote server response contains an error. FUOTA is canceled.”);
Serial.print(“HTTP request response: “);
Serial.println(http_line);
Serial.println(“Rebooting ESP32 due to FUOTA error”);
ESP.restart();
return;
}
}LISTING 7
This is the code to parse firmware size and validate the HTTP request response data type received.
/* Parse firmware size from HTTP header received from remote server */
if (http_line.startsWith(“Content-Length: “))
{
FUOTA_firmware_size = (size_t)atoi((search_for_string_in_http_header(http_line, “Content-Length: “)).c_str());
sprintf(verbose_debug_line, “Firmware size: %d bytes”, FUOTA_firmware_size);
Serial.println(verbose_debug_line);
}
…
if (http_line.startsWith(“Content-Type: “))
{
contentType_http = search_for_string_in_http_header(http_line, “Content-Type: “);
if (contentType_http == “application/octet-stream”)
{
is_firmware_valid = true;
Serial.println(“Data type corresponds to firmware data: application/octet-stream”);
}
else
{
Serial.println(“Data type doesn’t correspond to firmware data: it’s different from application/octet-stream”);
Serial.println(“Rebooting ESP32 due to FUOTA error”);
ESP.restart();
}
}LISTING 8
The firmware flash is verified, and the ESP32 is rebooted with new firmware.
/* Check if the entire firmware binary has been downloaded and flashed */
if (bytes_written_to_flash == FUOTA_firmware_size)
{
sprintf(verbose_debug_line, “Success: %d bytes written in Flash”, bytes_written_to_flash);
Serial.println(verbose_debug_line);
}
…
if (Update.end())
{
if (Update.isFinished())
{
Serial.println(“FUOTA process has successfully finished”);
Serial.println(“FUOTA total time: “);
Serial.print(calculate_time_difference(timestamp_FUOTA_init)/1000);
Serial.println(“s”);
Serial.println(“* ESP32 is going to reboot with updated firmware”);
ESP.restart();
}
}TESTS AND RESULTS
Now that the FUOTA process has been described in detail, it’s time to see it in action. First, the firmware boots, connects to the Wi-Fi network and starts monitoring the time since boot and the state of the FUOTA push button, as shown in serial messages on the Serial Monitor (Figure 3). Note the firmware version (V1.0) in the red box.
Next, the FUOTA push button is pressed. The FUOTA process starts, the firmware is requested to the remote server, and the ESP32 downloads the firmware and flashes it in its flash memory (Figure 4). After this, the total time spent in the FUOTA process is written to the Serial Monitor, which is in the red rectangle in Figure 4. It’s important to mention that I intentionally put this firmware in the remote server with V1.1 as the firmware version, to be able to see the difference when the ESP32 boots.
Finally, the ESP32 reboots with the new firmware flashed in its flash memory (Figure 5). Note that the firmware version is now V1.1, indicating that the FUOTA process has been finished successfully.
Firmware output before the FUOTA
Firmware output during and after the FUOTA
Firmware output after the ESP32 boots with the updated firmware
WHAT TO DO NEXT: IMPROVEMENTS ON ESP32 FUOTA
I have presented the basics and fundamentals of a Firmware Update Over the Air—what it is, how it works, and how MCUs and SoCs fit in the FUOTA process. I’ve also shown a practical example of how to conduct a FUOTA in an ESP32.
This article contains a big picture of the FUOTA process for the ESP32, and it should be seen as a starting point for those who want to implement and use the method in real-world projects. Keep in mind that a few more things related to security should be considered, studied, and implemented for a robust and complex FUOTA process, to ensure that firmware won’t be exposed to malicious agents. I suggest you spend some time reading and learning more about the following topics.
First, learn more about secure protocols to transfer firmware data and firmware requests between a remote server and an MCU. Such protocols preclude interception of the firmware data (firmware binary) by malicious agents, and replication of the binary to other devices (for cloning or reverse engineering purposes). The more of the firmware you hide, the safer your embedded system will be.
Second, consider choosing MCUs that support firmware encryption features, as the ESP32 does. These features make it harder for a malicious agent to intercept and clone firmware, even if the agent gains physical access to the MCU. Without knowing the encryption key, all the attacker can obtain are meaningless blobs of data.
Third, consider using a digital signature to ensure that the downloaded firmware really came from a reliable source. It will help to avoid attacks such as the “man-in-the-middle” attack, in which an attacker covertly intercepts, and possibly alters, data communicated between two devices.
REFERENCES
[1] Espressif software development kit
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/
[2] ESP32 documentation
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/partition-tables.html
[3] Arduino-ESP32 official support by Espressif
https://github.com/espressif/arduino-esp32/blob/master/libraries/Update/src/Update.h
[4] GitHub repository for the example project
https://github.com/phfbertoleti/ESP32_FUOTA_example
RESOURCES
Arduino IDE:
https://docs.arduino.cc/software/ide-v1/tutorials/arduino-ide-v1-basics
ESP-IDF:
https://docs.espressif.com/projects/esp-idf/en/latest/esp32/
PUBLISHED IN CIRCUIT CELLAR MAGAZINE • DECEMBER 2022 #389 – Get a PDF of the issue
Sponsor this Article
