Tools to Help Us Debug
This month, Bob continues his series on debugging embedded real-time systems. We will look at the kind of tools needed for debugging embedded systems that require us to plan ahead.
In 1975, I was designing (sort of) and debugging (mostly) systems built around the PDP-11 and the Intel 8080 (Figure 1). All of the code was written in assembly. Debugging tools were very primitive, and debugging was mostly done by running the finished (so far) code in the actual hardware.
If you only knew how hard it was to debug the 8080 microprocessor.
For the PDP-11 code, we had a dot matrix printer, the LA-36, where we could modify and dump some data. Digital Equipment Corp. (DEC) included what it called the Octal Debugging Tool (ODT) with its operating system. When the debugger was built with your software, you could put a break point instruction into your code where you wanted to stop. Once the software stopped, you could interrogate and change memory and processor registers. Upon stopping, the ODT actually replaced the break point trap instruction with the original code so that you could easily resume the software and then put the trap back. Amazing!
I am being silly. But at the time we thought this was awesome. Table 1 shows the full “capability” of the ODT. As impossibly primitive as it sounds, we were able to create some pretty complicated embedded control systems with it and with debuggers like it. [1]
DEC ODT capabilities
But what about today? What tools do I need to debug embedded systems today? This month I am going to look at the tools we need before we even start coding. Most of these “tools” were available to us in the 1970s and we relied heavily on them (as you can understand based on the ODT). In my opinion, they are needed just as much now as we needed them then.
EMBEDDED SYSTEMS SOFTWARE DEBUGGING TOOLS
If you only remember one thing from this article, remember this:
Debugging begins before the first line of code is written.
— ADVERTISMENT—
—Advertise Here—
Before a project is too far along, the software team needs to make sure that certain measures are taken in the hardware design to enable us to debug the software. Don’t fall for the trap that there will be no bugs that you can’t find in your design. Let’s look at some of the things we will need before we get started. Not all of these are needed on every project. But often they can be provisioned for at no recurring cost to the hardware.
SPARE GPIO, LEDS, AND PROVISIONAL HEADERS
Eventually, your software is going to be running without all the debug paraphernalia that’s normally included in your debug environment. Spare GPIO, a few LEDs, and provisional headers sometimes can provide just what you need to debug when everything is buttoned up tight and your debug tools are not available.
SPARE GPIO
Using spare General Purpose I/O (GPIO) pads is one approach. What? Your project doesn’t have any spare I/O? Don’t let your hardware designers eliminate spare GPIO. They will thank you later. “But this PIC microcontroller has just the exact amount of I/O and cost $0.03 less.” Don’t buy it. Put your foot down. You need spare I/O.
If you cannot get spare I/O, go looking for I/O that can be shared. For example, board I/Ds and board revs are only read once at power-up. After power-up, they can be yours. With most GPIO, an input can be made an output after you have read the I/Ds. Any input that doesn’t ever change can become spare after startup. Any spare D/A channels? A spare D/A can be used to provide a voltage output that can represent some state, error code or start-up sequence.
Once you have your “spare” GPIOs, make sure that they are brought out to a test pad or provisional I/O. There are great ways to connect to a board other than through an on-board connector. More on that later.
Let me give one example of what a spare GPIO pin can be used for when debugging real-time usage issues. If at some point in the software you have an idle loop, set the bit on while in the idle loop and off when something causes you to fall out of the loop. Using a storage scope you can measure the amount of spare real-time you have in your system. I am always surprised when designers tell me that they don’t know how much spare real-time they have. Interrupt latency and task real-time usage can also be debugged using this technique.
LEDS
If at all possible, provide at least one LED on the board. If necessary, make them provisional for development. You can provide the following indications with the LED to aid debugging: start-up progress; system health or heart beat; system mode; diagnostic codes; and so on. Pick essential milestones at start-up and provide a visual indication of each milestone with a code. Do the same with the system mode if appropriate. Finally, if the software detects a problem, flag it with a diagnostic code. Believe me, all of these will help you when the code needs debugging.
Not everything on the LED has to be interpreted by the naked eye. Make sure there is a pad you can access and connect it to a storage scope. The LED GPIO output can be used for visual and non-visual indications. Diagnostic codes, progress codes, and system mode can all be non-visual codes that can be read on a scope at the same time you are providing a heart-beat indicator. Pulse the GPIO very fast during the off time of the LED to indicate status or error codes.
PROVISIONAL HEADERS
If you don’t have resources for a debug LED (space, dollars, available outputs, or power), consider a debug board with a connecter-less interface like Tag-Connect (Figures 2 and 3). We have used Tag-Connect products [2] to keep the board costs at zero while providing a wide range of debugging interfaces.
— ADVERTISMENT—
—Advertise Here—
Tag-Connect connector
Tag-Connect Headers
DEBUG CONSOLE
This is one of the most critical pieces of hardware for software debugging. Preferably the debug console should be a serial port that can be easily written to and read from. With the debug console you can print out program status and data in real time. Warnings and alerts can be logged to the port. Most commonly, we would use a spare UART. If no spare ones are available, think hard about how you can share the debug console with an already used UART. Very often, it can be done. When the hardware did not have room for the debug console, RS-232 drivers and/or connectors, we would build a debug board with serial connectors and a connector-less means (e.g. Tag-Connect) to connect with pads on the target board. The hardware designers won’t spare a serial port? Grab some spare GPIO ports and bit-bang the console. There are plenty of off-the-shelf bit-bang algorithms for you to use.
Logging status, mode changes and events to the debug console connected to a PuTTY-like [3] interface will give you the ability to write everything to a log file not on the target.
JTAG PORTS (IEEE STD 1149.7)
If your processor supports it, you need to have the hardware support a JTAG interface. As we mentioned before, this can be a connector-less addition to the boards by providing pads for a header only used on engineering boards or using the Tag-Connect-type interface previously mentioned. JTAG technology [4] on today’s microprocessors and microcontrollers has replaced the need for an In-Circuit-Emulator (ICE). The JTAG standard has provided the definition for a five-pin interface for a debugger to access not only all of the external pins but also all of the internal registers of your device. It does this by a method called a boundary scan. With a JTAG interface you can also program your software.
A competing interface is the Serial Wire Debug (SWD). Although faster than JTAG and using only two wires, it does not use boundary scan technology and can only be used for debugging and programming. It is also only available on Arm processors.
SIMULATORS
Software simulators (Figure 4) can take many forms. I want to break them down into full, partial, and indirect.
Software simulators can take many forms. I break them down into full, partial, and indirect.
FULL SIMULATORS
I have used several full simulators over the years. I often used a Z80 simulator while developing machine control software that ran on the STD bus. [5] We used a full simulator for a PowerPC while working on a Full Authority Digital Engine Control. There are a lot of full simulators with powerful interfaces available for a wide range of processors. You can set break points, watch points, step through your code, and perform “real-time” performance checking (for example, count machine cycles through a path of the code).
The problem with any “full” simulator is that it doesn’t simulate your unique I/O. And with embedded systems software, I/O is everything. You can add-on to the simulator to simulate your I/O but that often is not worth the effort.
That said, simulators are great for debugging algorithms and even large chunks of logic-based software.
PARTIAL SIMULATORS
This is my own distinction and we have used this with great success over the years. It is probably best described by an example. We were developing a GUI-based time-and-attendance system for a client. The GUI was months away from being available. We chose a graphics library that was available for hardware that we did have (in this case, a PC), and were able to develop the GUI months ahead of actually getting the hardware. When the hardware was available, there were very few seams that needed to be ironed out.
We have used this in a number of scenarios successfully by taking advantage of software libraries that run on other existing hardware, giving us a significant leg up in development. This is especially helpful given the labor-intensive nature of the user interface design.
INDIRECT SIMULATORS
This is my own terminology. It works like this: If you are programming in a high-level language, why not develop large swaths of your code on a PC using something like Visual Studio? It is almost always a good idea to isolate your hardware interface from your core application algorithms. When you do that, it makes it easier to use a full simulator or to use indirect simulation. It is very important not to spend too much time developing these hardware simulators, so it is a balancing act. You don’t want to spend 50% of your time creating these I/O routines that simulate your hardware. But if you have a serial interface, why not debug the interface on Visual Studio until your hardware is ready? Note: you are not debugging the drivers, but instead things like the protocol. Or if you are using Berkeley sockets, why not get that software up and running on your PC? We have done this on a lot of large projects where the hardware was months behind in development.
DATA LOGS
My last bit of advice for preparing for debugging was not readily available to us in the early days. Debugging is greatly aided by the presence of a good set of system logs which will tell us (or try to tell us) what happened. For a more detailed take on what is needed for good logging, check out my article “25 Essentials to Embedded Systems Design” in Circuit Cellar’s 25th Anniversary Edition. [6]
But you might say, “I don’t have a file system on my device. How can I create a system log?” My advice is: Do everything you can to get a system log. Serial EEPROM’s are cheap. Under a quarter will get you 1k of log space. I know that is not much, but do whatever you can to get some log space. And then use it wisely.
SUMMARY
Debugging starts before you write your first line of buggy code. Taking these steps will help you solve a lot of extremely difficult issues downstream. Next time I will look at what you would normally think of as debugging tools, such as Source Code Debuggers; Logic Analyzers, and Serial Analyzers. Of course, only in thin slices.
— ADVERTISMENT—
—Advertise Here—
REFERENCES
[1] For more information about ODT for PDP-11’s running RSX11M, check out the 1975 ODT manual ·
http://bitsavers.informatik.uni-stuttgart.de/pdf/dec/pdp11/rsx11/RSX11M_V2/DEC-11-OIODA-A-D_ODT_Dec75.pdf
[2] Tag-connect products were a wonderful addition to almost all of our products for over 10 years. https://www.tag-connect.com/
[3] PuTTY is a great free terminal emulator giving you an SSH and telnet client. I would recommend it over all of the competitors. putty.org
[4] The JTAG specification has become an IEEE standard. You can purchase the standard here: https://ieeexplore.ieee.org/document/5412866/figures#figures
[5] We used STD bus products for over 20 years in a number of real-time machine control applications. See https://en.wikipedia.org/wiki/STD_Bus for more information.
[6] Although many years old, there is still a lot of wisdom in the pages of the 25th Anniversary Issue of Circuit Cellar https://cc-webshop.com/collections/books-electronics/products/cc25-anniversary-issue
RESOURCES
Tag-Connect | www.tag-connect.com
PuTTY | putty.org
PUBLISHED IN CIRCUIT CELLAR MAGAZINE • OCTOBER 2022 #387 – Get a PDF of the issue
Sponsor this Article
Bob Japenga has been designing embedded systems since 1973. From 1988 - 2020, Bob led a small engineering firm specializing in creating a variety of real-time embedded systems. Bob has been awarded 11 patents in many areas of embedded systems and motion control. Now retired, he enjoys building electronic projects with his grandchildren. You can reach him at
Bob@ListeningToGod.org
