Performance Bottlenecks in Embedded Linux Solutions

Analysis, Identification, and Mitigation

Good performance is a requirement for every technology, and system designers rely on operating systems to ensure fast and smooth transitions in critical applications. Fortunately, Pedro writes, the embedded Linux OS offers ways for finding, analyzing and mitigating performance bottlenecks so embedded systems can deliver the speed and efficiency that end users expect.

The world of technology today seems to obey the famous quote of philosopher Heraclitus of Ephesus, who said, “The only constant is change.” In terms of innovation, technology is one of the most dynamic: Rapidly evolving, and changes can seemingly occur overnight. There’s one exception, however: Good performance. This is a constant requirement for every single technology. Technology exhibiting poor performance won’t be used extensively, end users won’t like it, and it will quickly die.

For embedded operating systems (OS), good performance is critical because the software often handles critical solutions that must be fast and smooth, like safety controls in motor vehicles. This is also true for embedded Linux.

This article explores some methods and tools for analyzing the performance bottlenecks of embedded Linux—helping to identify them and giving some insights on how to mitigate them.

PERFORMANCE BOTTLENECKS

Unfortunately, the term, “performance bottlenecks analysis” isn’t very precise, as long as it can comprise a wide range of definitions, from user experience metrics to specific hardware metrics. This means that first, you must determine a common base for the analysis, which corresponds to defining which topics (or points of view) will be taken in consideration.

The common base for performance bottlenecks analysis focuses on the three of the most important resources in any embedded system: CPU (central processing unit), random access memory (RAM), and swap memory (the part of a computer’s hard drive where data is stored temporarily, when the RAM is full). To better detail this analysis, the following topics will be covered:

• CPU usage analysis, for both specific processes and general system usage
• RAM usage analysis, for both specific processes and general system usage
• Swap memory usage analysis
• Memory thrashing analysis
• Memory leak analysis

CPU USAGE ANALYSIS

Proper CPU usage in embedded systems running Linux—in fact, for any embedded systems solution—is crucial for maintaining overall system performance or optimizing power consumption. Excessive or prolonged high CPU usage not only increases power consumption and raises the temperature of the SoC (system on chip) or SiP (system in package) but can also cause thermal throttling. Thermal throttling leads to general embedded Linux slow-down, significantly affecting its efficiency and user experience, as shown in Figure 1.

When an embedded Linux solution shows signs of sluggishness and bad performance, one of the first things to do is to check if the CPU usage (both per-process and overall cases) is consistently high over time. If it is, then one of two scenarios is likely to be true:

The SoC or SiP being used has CPUs that are undersized for the intended solution. It means the SoC or SiP cannot deliver the needed processing power.

The SoC or SiP has sufficient processing power for the intended solution, but due to development issues, bad programming practices, and/or errors, certain processes are demanding far more CPU resources than expected.

In the first case, there’s no easy way out; the solution is a hardware upgrade to a more capable SoC or SiP, with more powerful CPUs, since the current hardware lacks the required processing power. In the second scenario, however, optimizing software design or addressing any bugs to the process source-code responsible group, enterprise, or developer can significantly improve CPU usage, resulting in better overall performance.

But how can CPU usage, both overall and by individual processes, be measured to determine if optimization is possible? Several measurement tools are available on Linux, one of the most common of which is the “htop.”

The htop is an interactive system monitor, process viewer, and manager, tailored for Unix-based systems, which is the case of Linux, a derivation from Minix, which is based on Unix. Built as an alternative to the Linux utility called “top” (Table of Processes), the htop provides similar functionality but with enhanced flexibility, a nicer appearance, and customization options for viewing and managing system processes. The htop tool provides a real-time, dynamic view of various Linux system metrics, including CPU usage, RAM usage, swap usage, process uptime, virtual memory per process.

Specifically for CPU usage, the htop allows visualization of the following features:

• Instantaneous individual CPU usage percentage for each available CPU, (highlighted in green in Figure 2).
• CPU usage per process, (shown in orange in Figure 2).
• Average overall CPU load over time, known as “Load Average,” (highlighted in light blue in Figure 2). This includes average CPU usage over the last minute (first Load Average value), the last 5 minutes (second Load Average value), and the last 15 minutes (third Load Average value).

When using the keyboard keys from F3 to F9 in the htop tool, it’s possible to search for specific processes (which is especially useful for tracking just one process among the many an embedded Linux OS system may have), filter the process view, and kill specific processes.

The htop tool is therefore highly useful for monitoring CPU usage on embedded Linux OS, providing insights into both instantaneous CPU usage (overall and per process) and the average CPU usage over time.

Another great tool for analyzing CPU usage per process is called “ps” (Process Status). The ps tool’s primary function is general process monitoring, displaying metrics such as CPU usage percentage per process. Additionally, it allows the developer to check which processes are actively running, giving a comprehensive view of processes execution on an embedded Linux OS at any moment. An example output of ps tool is shown in Figure 3.

A third useful tool for analyzing CPU usage on embedded Linux is called “sar.” The sar (System Activity Reporter) monitors overall activity of each CPU present on the solution SoC or SiP, providing metrics such as percentage of CPU usage, CPU idle time, CPU IOWait, and much more. This tool is an excellent choice for assessing CPU load and obtaining a comprehensive view of CPU usage by the embedded Linux system.

It’s possible to configure the sar tool, for setting the monitoring interval (time between CPU features measurements) and specifying which CPUs the tool will monitor. For example, to monitor the first CPU available on a given SoC or SiP, with a 500ms interval between CPU features readings, the following command can be used:

sar -p 1 500

From a practical standpoint, here are some strategies for investigating CPU usage in embedded Linux OS:

Step 1: Check the Load Average (over the last 1, 5, and 15 minutes) to see if the CPUs are undersized for the intended solution. If so, proceed to Step 2.

Step 2: Use the individual CPU usage percentage to check if any process or application is consuming more CPU than expected (or maintaining a consistently high usage). If none are found, the overall set of software in the solution may exceed what the SoC or SiP CPUs can support. In this case, consider optimizing for IOWait to help minimize performance issues. If a process with anomaly high CPU usage is found, move to Step 3.

Step 3: If the process using excessive CPU is one you developed, investigate possible optimizations regarding too many threads being used, waste CPU time on unnecessary code blocks, and so on. If it is third-party software, look for updates, or even consider replacing it with a more efficient alternative.

Another good approach is to rewrite processes (obviously, processes you or your teams are responsible for) in a more efficient programming language. In many cases, processes developed using higher-level languages require more runtime and processing, leading to increased CPU usage over time. It could be a good idea to assess whether, for certain CPU-intensive processes and applications, using a language with a shorter runtime might be beneficial. See Table 1 where a programming language benchmark [1] is shown, highlighting that the Python programming language can be almost 76 times slower than C.

RAM USAGE ANALYSIS

Efficient RAM usage is essential for optimal performance in embedded software, because during system active use, RAM stores all applications, processes, and essential elements needed for the operating system (in this case, embedded Linux) to run and use—with access times far faster than mass storage memory (hard disks, for instance).

Excessive RAM usage by certain processes can lead to a situation where the available (or free) RAM is insufficient to handle all processes, temporary data (buffers), and other essentials needed for the embedded Linux OS to operate smoothly. This forces the OS to manage RAM through memory swapping or even force processes termination to free memory, leading to a significant performance reduction. In extreme cases, this can result in “memory thrashing,” where the time spent managing memory paging becomes equally or more significant than the time spent running necessary processes, themselves, causing severe system sluggishness. The topics of memory swap and memory thrashing will be discussed in detail later in this article.

When an embedded Linux-based solution exhibits bad performance, alongside checking CPU usage, one of the steps should be to assess whether RAM usage by the solution’s processes is excessive (more than expected, considering an analysis of RAM consumption over time). If so, this indicates a RAM usage issue, which could stem from excessive RAM consumption by certain processes due to software design issues, or even a memory leak (a condition that will also be covered in detail later in this article).

All Linux tools discussed so far—the htop, ps, and sar tools—can also be used to monitor RAM usage. In the htop tool, each reported process includes a percentage of memory used (labeled MEM%), as well as total memory in use and swap usage. Thus, the htop tool is highly useful for investigating the instantaneous memory usage of one or more processes on embedded Linux OS. In Figure 4, it’s possible to see the RAM usage percentage per process (highlighted in green) and the overall RAM and swap usage (highlighted in blue).

In addition to the htop tool, another great tool for analyzing RAM usage on embedded Linux OS is called “free.” What sets the free tool apart from other tools is its ability to provide an overall view of memory usage in embedded Linux OS, including total, free, and used RAM, as well as memory shared across processes in Embedded Linux OS. Through its metrics, it’s even possible to identify signs of memory leaks and memory thrashing—both of which are discussed a little later.

The following are strategies to help mitigate high RAM consumption in embedded Linux OS:

• Use ZRAM. ZRAM acts as a RAM drive, where all data is stored in a compressed format and is automatically decompressed when read. Since RAM access times are significantly faster than disk access, using ZRAM as memory swap can help minimize the slowdown caused by swap operations.
• Adjust the swappiness value (typically in /proc/sys/vm/swappiness file). This parameter controls how aggressively Linux performs swapping, allowing you to minimize swap operations within a given timeframe.
• Identify possible memory leaks. Memory leaks lead to high RAM usage and thus excessive swap and memory thrashing. If a process you or your team has developed shows increasing RAM consumption over time, you can use tools such as “valgrind” to locate memory leaks and address them accordingly.

SWAP MEMORY USAGE ANALYSIS

First, let’s define what memory swap is. Briefly, memory swap is an operation performed by the operating system (including embedded Linux) to move less frequently used data from RAM to a mass storage (such as SSD, micro-SD card, HDD, and so on). This frees up space in RAM for general operating system use, and allows data stored on mass storage to be reloaded into RAM if one or more processes need them. Thus, memory swap helps maintain a healthy amount of available RAM, resulting in faster operation of frequently used processes.

That said, memory swap is a must-have operation for any operating system, especially those that run in SoC or SiP with RAM limitations. This is common in the case of embedded Linux systems, since adding more RAM increases hardware costs, and large amounts of RAM should not be expected in embedded systems that need more and more competitive costs.

Memory swap comes at a cost, however—it causes general sluggishness. The speed of reading, writing, and communicating with RAM is much faster than with mass storage, especially HDDs. Each swap operation is limited by the mass storage’s overall speed (read, write, and communication), making it extremely slow from the OS’s perspective. Then, it doesn’t matter how fast a CPU can be; if the data needed for processes is stored in a slow-access memory, everything will slow down too. Figure 5 shows why memory swap to a mass storage memory is slow.

To check the amount of swap memory used on embedded Linux, the quickest methods are to use tools like htop or free, as discussed previously in this article for CPU and RAM usage analysis. Manually or automatically monitoring swap usage over time can give a big picture on how aggressively RAM is being consumed on a specific embedded Linux OS-based solution.

Because memory swap is a slow operation, avoiding memory swap is recommended to achieve better overall performance on embedded Linux OS-based solutions. To minimize memory swap occurrences in embedded Linux OS, it’s important to understand its main causes. The three most common causes of frequent memory swap in embedded Linux OS are low amount of available RAM, unoptimized memory swap configuration, and high RAM usage by specific processes.

Low RAM Amount Available: When the amount of available RAM is below the solution RAM memory requirements, memory swap tends to occur frequently, significantly slowing down the operating system as a whole.

Unoptimized Memory Swap Configuration: Memory swap is triggered once free RAM reaches a certain threshold. In embedded Linux OS, this is controlled by the swappiness parameter / configuration, which typically can be found in the /proc/sys/vm/swappiness file. This file contains a number ranging from 0 to 100, indicating the percentage of free RAM at which memory swap should activate. (Setting it to zero disables swap and setting it to 100 allows frequent memory swap operations.) Thus, higher swappiness values (closer to 100) mean memory swap will begin even with a considerable amount of available free RAM. This can be unnecessary in many real-world scenarios, causing a general slowdown in embedded Linux OS. For instance, if swappiness is set to 80, swap begins when RAM is at 20% free. Moderate swappiness values (around 60) lead to good balance in swap usage, without causing significant slowdowns to the operating system.

High RAM Usage by Specific Processes: Occasionally, due to suboptimal development choices, specific processes may allocate more RAM than expected, consuming more of the system’s RAM and increasing memory swap usage. This situation often arises when a solution developed for a personal computer or server environment (likely with more RAM available) is ported to embedded Linux OS, without taking in consideration the amount of RAM available in target hardware.

MEMORY THRASHING ANALYSIS

Cutting to the chase, memory thrashing is a direct consequence of using too much memory swap. As previously discussed, the bottleneck with memory swap is the slower read, write, and communication speeds associated with mass storage memory used for storing memory swap data.

When embedded Linux OS reaches a state where free RAM is so limited that constant memory swapping is required, frequent interaction with mass storage ensues. Since this interaction is slow, the system may reach a point at which it spends more time performing memory swaps than using the data read and written in memory swap operations, leading to an extreme slowdown in embedded Linux OS. This situation is commonly known as “memory thrashing.”

To put it in perspective, imagine digging a deep hole using only a spoon. The spoon (representing RAM) can carry only a small amount of material such as data, buffers, so the person digging (representing the operating system) must make frequent trips to a storage container (mass storage) to empty the spoon (memory swap) and then resume digging. Given the spoon’s limited capacity, this emptying process takes significant time relative to the main task of digging, leading to excessive time spent swapping or memory thrashing.

To diagnose memory thrashing, certain RAM usage metrics in embedded Linux should be monitored closely. These metrics are available through the same tools covered so far in this article—the htop, sar, and free tools.

Some indicators of memory thrashing in embedded Linux OS are noted below:

• High swap memory usage.
• Swap memory usage is increasing substantially over time.
• Rising I/O wait time (known as IOWait in embedded Linux). The IOWait reflects the percentage of time one or more CPUs spend waiting for responses from an I/O device, such as mass storage memory. Therefore, the more memory swap occurs, the greater the IOWait value will be.

Since memory thrashing is highly undesirable in embedded Linux OS, it’s essential to minimize memory swap. This can be done by monitoring and mitigating the causes of memory swap discussed above.

MEMORY LEAK ANALYSIS

In terms of ensuring good performance, a memory leak is one of the worst enemies that embedded Linux can have. A “memory leak” is a serious flaw in software, where memory allocated dynamically by the software is not released after use, rendering that portion of RAM unavailable for any other program. If this issue occurs in a frequently-called routine, it will quickly exhaust the system’s free RAM, leading to severe performance issues—forcing memory swap, causing memory thrashing, and so on—and potential system failures. In extreme cases, it may even crash the entire system, requiring a restart.

A classic case of a memory leak is when a C program allocates memory dynamically using the malloc() function but fails to release it afterward with the free() function. When this is done one hundred times a second, here comes one of the most common embedded software developer’s nightmares.

To detect memory leaks in embedded Linux, it’s recommended to monitor RAM usage periodically (system-wide or by individual processes), either manually or automatically. This can be done using tools discussed previously: the htop, ps, and free tools. If RAM usage of a specific process steadily increases over time, it suggests a memory leak.

CONCLUSION

This article concerns techniques and tools for analyzing the performance of embedded Linux, from the perspective of CPU and RAM use.

It is possible to perform an analysis under these aspects to determine whether processes are using more computing resources than expected. In this way, the developer will be able to understand the possible reasons why the performance of embedded Linux in a given solution is lower than what is expected, and also determine which processes are responsible. 

REFERENCES
[1] https://thenewstack.io/which-programming-languages-use-the-least-electricity
[2] https://phoenixnap.com/kb/swap-memory

SOURCES
https://man7.org/linux/man-pages/man1/top.1.html
https://www.embedded.com/performance-analysis-of-linux-based-embedded-systems-part-1
https://man7.org/linux/man-pages/man1/ps.1.html
https://man7.org/linux/man-pages/man1/sar.1.html
https://www.ic.unicamp.br/~rafael/materiais/valgrind.html
https://monovm.com/blog/what-is-htop-and-what-does-it-do

PUBLISHED IN CIRCUIT CELLAR MAGAZINE • JANUARY 2025 #414 – Get a PDF of the issue

Sponsor this Article