Tools Needed to Debug On the Go
This month, I continue my series on debugging embedded real-time systems with a discussion on some more of the generic tools needed for this task. Previously, I discussed debugging tools that require planning ahead. Now I’ll look at the tools that can be purchased as you need them.
Sometime in the late 1970s, I attended a tradeshow dedicated to developing industrial controls. Included at the show were tools for creating embedded systems. I had been developing embedded systems for about five years using pretty crude tools to design and debug (not to mention that my own skills were pretty crude). As I mentioned in my last article in Circuit Cellar’s October issue #387, one tool I was using was the Octal Debugging Tool (ODT). It was text-based and worked at the assembly language level. As I rounded the corner of the tradeshow, I saw a graphics screen displaying C code with break points and watch points for debugging—a high-level source code debugger! I was star-struck. Source-level debugging in C on a graphics terminal was a thrilling prospect. I could imagine how productive I might be with such a tool. But they were expensive—I mean really expensive. Yet within a few years I convinced my boss that the team needed some graphics terminal source-level debuggers. We settled on buying four HP64000’s (Figure 1). The cost seemed astronomical. My boss asked, “Are you sure you need this?”
It turned out that we did need them, and we went on to design some amazing stuff using them. But my boss never forgot the price tag. We had these units in our lab for a team of 12, so we had to carefully schedule our time and do shift work. One day, a designer who was scheduled to use one of these put a note on it that read, “This HP64000 is free.” My boss exploded (he really was a good guy and good boss) and demanded that the note be replaced with, “This HP64000 is available.”
HP64000
Well, we’ve come a long way since then. Today a plethora of tools are available from many sources. I will not be evaluating individual implementations of these tools in this article. Good reviews abound online. When you purchase something that you have to assemble at home, the instructions usually tell you what tools you need, not their make and model. For example, instructions don’t say that you need a ½ inch DeWalt cordless drill—they say you need a drill. That’s what I’m going to do this month. And I will focus on tools that address typical errors—the bugs that bug us.
Table 1 shows all the bugs we have discussed in the previous articles, and the suggested appropriate tool for debugging each of them.
Tools needed for different bugs
Tools for Debugging Embedded Systems
A top-notch compiler: Wait! That isn’t in Table 1. That’s because I forgot to list one of the most frustrating bugs of all time in my previous articles: compiler bugs. I cannot tell you how many compiler bugs I ran into over my 50 years of debugging real-time embedded systems. The first bug was with a JOVIAL compiler while designing a fly-by-wire flight control for NASA’s X-Wing (Figure 2). JOVIAL was an offshoot of the pioneering language ALGOL (developed by a joint team of European and American computer scientists in 1958). JOVIAL was developed from ALGOL 58 specifically for embedded systems. Jokingly referred to as Jules’ Own Version of ALGOL, the compiler we used was clunky and full of bugs. Our code was complex, and the compiler bugs made our job hard.
NASA’s X-Wing
Fast-forward 50 years and compiler bugs are few and far between. But they can still be present in either the compiler or the runtime library that the compiler uses. Top-notch compilers provide an immense amount of static analysis of your code and can provide all kinds of warnings about potential bugs. In fact, you might be tempted to turn off the warnings as they can overwhelm you. I have found that compiling with all warnings on throughout the life cycle of the project is like wearing bug repellent. It will prevent more bugs than you might expect.
— ADVERTISMENT—
—Advertise Here—
Static analysis tool: The first static analysis tool that I used was called “lint.” Named after the lint trap in your dryer, it was initially developed in Bell Labs in the late 70s to help find porting issues with Unix, the predecessor of Linux. Over the years, a lot more static analysis has been incorporated into compilers. But as the static analysis algorithms have evolved and become more complex, taking longer to run and expanding across files, it makes sense as a strategy to have a separate tool to dig down into our code. In addition, a static analysis tool can run its analysis across multiple files whereas our compilers are looking at only one file at a time.
Table 2 outlines what I would look for in a quality static analysis tool. Wikipedia has a list of available static analysis tools that is worth looking into [1].
Runtime memory error checker: Static analysis can only go so far in detecting stack overruns, memory leaks, buffer overruns, and pointer problems. You can deepen your bench during testing with an add-in memory error checker to build into your code that can be disabled during production or even compiled out. Some operating systems provide this, but most RTOS’s that we used did not. Even with Linux, we needed more sophisticated add-in memory checkers than were provided.
The first such tool that we employed was called Memcheck (sadly it went out of business) and it helped us in debugging more than almost any other tool.
Table 3 provides my wishlist for a Runtime Memory Error checker.
My wishlist of features in a static analysis tool
My wishlist of features in a memory checker tool
Storage oscilloscope: The tried-and-true digital storage oscilloscope is a wonderful debugging tool. One of our projects required an output to be accurate to within one millisecond. We could set up the scope in envelope mode and watch the output throughout our testing and even over the weekend. Any output that was out of our tolerance band would show up. Fixing it was another story.
We also used the scope to analyze real-time usage. By instrumenting a periodic thread, such as an interrupt, at the start and the end, you could measure the percent of real-time that each thread took.
Often, I found that many of our designers were not comfortable using a scope. I suggest you get a good one and learn how to use it. You will find it invaluable in debugging real-time systems.
Debugging state of the art hardware requires a high-speed scope which can be very expensive. But in most cases with software, you do not need such high-end scopes. Equipping every designer with a much lower speed scope is highly feasible now. I’m retired and a $20,000 scope is not in my budget, but I have one that I purchased for under $100. I wouldn’t use it to try to debug a memory interface with an Arm processor, but it works great for debugging slower signals.
— ADVERTISMENT—
—Advertise Here—
Serial port analyzers (RS-232, RS-422, SPI, I2C): I mentioned earlier in this article that the HP64000 was a great tool for debugging. One of our customers had a serial analyzer that used the same hardware as the HP64000. We had previously attempted to use our scope to debug serial interfaces. This customer’s serial port analyzer opened my eyes to the power of debugging serial interfaces with these tools. Now we use serial analyzers for all our serial interfaces, and they’re cheaper and easier to use. They can act as passive monitors, or they can be programmed to simulate one of the active nodes on your serial interface. A wide variety of inexpensive systems are available with excellent software. Most support a wide variety of serial protocols. We used the Aardvark I2C/SPI (Figure 3).
Aardvark erial Analyzer
Ethernet/TCP/IP analyzers: As more and more of our designs became interconnected, Ethernet analyzers were available for analyzing and debugging Ethernet traffic. These are often called packet sniffers. Like serial analyzers, they can passively sit on the network and break down the vast array of minutiae that comprise a TCP/IP packet. During the early days of our Internet-of-Things projects, we used these to measure the amount of data we were using. Many of our early designs were allowed only one megabyte of over-the-air data per month. Since we couldn’t analyze the cell traffic, we would send the data over Ethernet and measure the packet size, the frequency of retries, and so on.
Sometimes we used an Ethernet analyzer to debug aberrant behavior on the web that either our embedded device or the server were generating. For this we used the free Ethernet analyzer Ethereal, which later became Wireshark [2].
Profiler: Early on in our life as a company, we used Borland tool chains. They had a software profiler, Turbo Profiler [3], that was very helpful in debugging real-time issues. Profiling measures items such as the frequency and duration of function calls, the complexity of the structure of the software, and the usage of particular instructions. Profilers were clunky back in the late 80s, but today a wide range of profiling tools are available for a wide variety of languages. It was an essential tool in our tool kit.
Thermal Gun: What? A thermal gun [4]? Isn’t that used by the hardware people? If you’re nice, they might let you borrow it. And no, we’re not going to check the foreheads of the team with it. We used high-end units on occasion to find places in the code where certain ICs are stressed, which can indicate a problem in the way we’re using the chip. They also came in handy when we wanted an independent means of measuring current draw on extremely low power applications—a little bit of current can affect the surface temperature of the processor. And they provided an easy way to see if overall real-time usage changed. They are versatile, and have many other potential uses I’m sure you can come up with (Figure 4).
Fluke Infrared Thermometer
Conclusion
The vast array of tools available for debugging embedded systems can be overwhelming. And we can get sidetracked exploring our new toys—that is, tools. But having a reliable set of debugging tools in your tool chest is invaluable. I hope this primer has been helpful, but of course I have only pointed to a few trees in the forest of options.
Next time, we’ll look at some general guidelines to follow in determining whether a bug is in the hardware or the software.
REFERENCES
[1] https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis
[2] https://www.wireshark.org/
[3] https://en.wikipedia.org/wiki/Borland_Turbo_Debugger
[4] https://www.fluke.com/en-us/products/temperature-measurement/ir-thermometers
PUBLISHED IN CIRCUIT CELLAR MAGAZINE • DECEMBER 2022 #389 – Get a PDF of the issue
Sponsor this Article
Bob Japenga has been designing embedded systems since 1973. From 1988 - 2020, Bob led a small engineering firm specializing in creating a variety of real-time embedded systems. Bob has been awarded 11 patents in many areas of embedded systems and motion control. Now retired, he enjoys building electronic projects with his grandchildren. You can reach him at
Bob@ListeningToGod.org
![Figure 1 The heart is actually two pumps in one. The right side is responsible for pumping blood into the pulmonary circuit, which transports blood to the lungs, where it picks up oxygen and delivers carbon dioxide for exhalation. The pulmonary circuit then returns blood back to the left side of the heart. The left side is responsible for pumping blood to the systemic circuit, which transports oxygenated blood to virtually all the tissues of the body. The systemic circuit then returns deoxygenated blood and carbon dioxide to the right side [1].](https://i0.wp.com/circuitcellar.com/wp-content/uploads/2022/08/383_Bachiochi_Figure_1-e1659683125950.jpg?resize=145%2C100&ssl=1)
