Something from Nothing
Debugging an embedded system can be difficult when you’re dealing with either a simple system with few pins or a complex system with nearly every pin in use. Stuart provides some tips to make debugging such systems a little easier.
Debugging a microcontroller system can be difficult. Things don’t work right and it often isn’t even clear why. Was something initialized wrong? Is it a timing issue? Is there conflicting use of shared resources?
Debugging is more complicated when there are limited resources. If all the processor pins are used, what do you connect to? How do you get debug information out of the firmware so you can see what is going on?
This article isn’t about debugging when you have Ethernet, USB, and Bluetooth interfaces available, or when you have a full-speed emulator. This is about debugging when there aren’t many resources available—simple systems with few pins, or more complex systems with nearly every pin already used for something.
POSTMORTEM VS. REAL-TIME DEBUGGING
There are two general ways of debugging an embedded system. One is postmortem, looking at the state of the system after it has failed or after it has stopped at a breakpoint. The other is real-time, debugging while the system is running. Each has its own place and its own set of challenges.
Generally, the two methods use different debug techniques. A postmortem debug happens when the motor is stalled, the software can’t recover, and you have no idea why it happened. You want to know the system’s state and how it got there. Setting breakpoints is a method used in postmortem debugging; you stop the system and look at the static state after a particular point in the code is reached.
Real-time or active debugging is more appropriate for looking at timing issues, missed interrupts, cumulative latency issues, and cases where the system just does occasionally does something strange but doesn’t actually stop. Real-time debug can tell you how the system got into the state that you are trying to analyze using postmortem methods. If you can capture enough information while the system is running, you have a chance to turn a real-time problem into a simpler static post-mortem analysis.
— ADVERTISMENT—
—Advertise Here—
UNIVERSAL DEBUG SOLUTION
An asynchronous serial port may be the most common debug tool used in the embedded world. Most microcontrollers have at least one serial port built in. The serial port has limitations. Its speed is limited and it requires level translation to connect it to the RS-232 voltage levels of a PC.
In many cases, you might not want to put the RS-232 driver on your board. You don’t want to use the space required by either the IC or the RS-232 connector, especially for something that is only used while debugging. One way I’ve solved this problem is shown in Figure 1, which depicts just a Maxim Integrated (or Texas Instruments) MAX3232 RS-232 driver IC connected to a DE9 connector. The other side connects to a four-pin header. This is connected via a cable to the embedded system to be tested. This allows the embedded system to have just the four-pin connector wired to the microcontroller serial port pins, power, and ground. You plug in the external circuit when you need to debug and unplug it when you are done. There is nothing special about this circuit, it is exactly the same as you might put on your microcontroller board. Except you don’t need the space on your board for this. The circuit takes power from the microcontroller board via pins 1 and 4 of the four-pin Molex connector. The connector indicated is polarized so you can’t plug it in backward. I’ve standardized on this in my embedded systems at least where the serial port is used for debug or download.
This is the schematic for a serial port RS-232 driver. It’s a standardized circuit that plugs into a header on the board to be debugged.
Although I used a connector with 0.1” centers on the interface board, there is nothing to prevent you from using a 2 mm or 0.05” connector, or even a row of pads at the edge of the board being debugged. You just have to make a cable that has the Molex connector at one end and whatever you need to match your embedded board at the other end. You can keep the driver board in your toolbox, put the connector on your embedded system boards, and you have it when you need it.
You can house the board into a plastic project box. In one case, I built one on a narrow piece of perforated project board, and covered the entire thing in heat shrink tubing. It has the right-angle Molex connector on one end, and a short cable with the RS-232 connector on the other end. I keep that one in my desk drawer at work.
USING THE SERIAL PORT
The traditional way of debugging with a serial port is to send ASCII messages and commands back and forth. This is very effective for postmortem debugging; you can write a small monitor that lets you examine memory or view the state of the registers. You can use a serial port program to display the ASCII text and the debug information is easy to read.
The drawbacks to this approach for real-time debugging are that sending a message comprising multiple ASCII bytes can significantly slow system operation. And the need to wait for the serial port to be ready to accept new data limits how much debug information can be sent and how quickly it can be updated. Even at high data rates such as 115,200 bps, multibyte messages can take significant time. And the performance impact of sending long strings of bytes can change system timing enough to make problems go away—or they can create entirely new problems unrelated to the problem you want to solve.
ANOTHER APPROACH
There are some other ways to use a serial port that are more suited to real-time debugging. One is to send a continuous stream of bytes to the host PC. The bytes could be either individual 8-bit bytes or two could be concatenated to make a 16-bit word. Each bit in the byte (or word) represents an active task or the state of a sensor or a fault condition or about anything you can represent with a single bit.
Since the data is binary, not ASCII, you can’t see it with a normal terminal program. To use this approach, you would write a program for the PC that captures the serial port data in binary form. By storing the last 1,000 (or 10,000 or 100,000) bytes, you have a running history of what is happening, and when it fails you have a record of the events and timing leading up to the failure. This sort of tool is very useful for debugging failures that occur long after the triggering event. I had to solve a problem like that once in a system that moved items along a track. A feed error couldn’t be detected until much later, when it resulted in a spacing error. If you do this, you will end up with a lot of data that may be hard to decipher. It may be useful to write a simple program in Python or something to parse the bytes into meaningful messages or to look for specific illegal conditions such as a task that shouldn’t ever run twice without a second task executing in the middle.
SENDING STATE VALUES
You can also send state values. You could have, say, a string of 5 bytes, each representing the state of some process or measurement or other value. This string is transmitted continuously and captured in the PC, and it’s then analyzed later for a history of the failure. Obviously, you get more information this way, but you don’t get it as frequently as when you assign meaning to each bit in a byte. And the more bytes you try to pack into each message, the greater the potential impact on system performance.
— ADVERTISMENT—
—Advertise Here—
The advantage of these approaches is that they make more efficient use of the serial port, and they minimize impact on throughput. Any real-time serial port debug method is going to use up some of the available CPU cycles. Encoding the data as binary instead of ASCII minimizes that.
TWO APPROACHES FOR TIMING
There are two ways to trigger sending a real-time serial message. You can send periodically or you can send when something changes. Both methods offer some advantages and disadvantages.
Say you have a microcontroller with a timer that you have programmed to produce an interrupt at 1 kHz. If you program the serial port to at least 19,200 bps, you can send a byte at every interrupt. The bits of the byte are set by the various processes that you want to monitor and the combined byte is transmitted at each time the 1-kHz interrupt occurs. The result is a string of bytes with implicit time information since each byte is 1 ms from the previous and next bytes (with some jitter due to latency). The drawback is that if an event happens and then goes away between transmissions, you will miss it.
The other way to do this is to transmit any time a state changes. The various processes that set bits in the debug byte will call a common routine to set or clear a bit. If this results in a change, the common routine will transmit the byte if the transmitter data register is available (empty). This lets you capture all state changes unless two occur in one transmission window. But you give up the implicit timing information. If you use this approach, you should program the UART to operate at the maximum data rate possible to minimize the chance that the transmit register will be full when a state changes.
NO SERIAL PORT?
What if you don’t have a serial port? The serial port pins might be needed for something else. Or maybe your microcontroller doesn’t have a serial port. Can you still get debug information? You can, of course, implement a serial port on any spare pin by generating the timing in software. But in most applications this would really only be suitable for postmortem debugging, dumping a mass of data after everything has stopped. It isn’t very practical for real-time debugging.
One approach to using a single pin is to periodically shift out a series of bits that represent the current state of the system. You can view this on an oscilloscope. You might miss transient things that come and go very quickly, but you can get an idea of what is happening, what tasks are running, and what conditions have occurred. Figure 2 shows how such a waveform might look on an oscilloscope. There is a start bit that you can use to trigger the scope. Then there is a bit for each condition you want to monitor. You can mix the state of active tasks with the state of conditions such as faults. In Figure 2, there are only 5 bits of data after the start bit. The point being that you aren’t limited to byte-wide values. You can send only as much information as you are interested in.
This is how to use a single output bit to generate a 5-bit status word. This would be displayed on an oscilloscope.
To use this, you would send the serial data on a regular basis, such as on a timer tick, and you would shift it out as fast as the microcontroller can. Although this approach is a variation on a UART implemented in software, it runs at whatever rate the microcontroller is capable of setting and clearing bits, so it is very fast. Typical C code to implement this would look something like Listing 1.
Listing 1
Code for single-pin status output
void SendAByte(ByteToSend)
{
uint8 counter ;
OutputBit1 ; // Start bit = 1.
// Send the byte, LSB first.
for (counter = 0; counter < 5; counter ++)
{
if ((ByteToSend >> counter) & 1) OutputBit1;
else OutputBit0;
}
OutputBit0 ; // Ending value = 0
}In that function, OutputBit0 and OutputBit1 would be macros or functions that set or clear the output on whatever port pin you have available. Typically, you will need to pad the zeros or ones with NOPs or some other delay to make the zero/one time the same. You can make this even faster, at the expense of code size, by using inline code instead of a loop. That avoids the shifting and counter manipulation steps. Those are easy to write in C, but they can use a lot of CPU cycles.
LOADING MEASUREMENT
The single output pin can also be used with an oscilloscope to monitor throughput. Say you have an idle loop that continuously checks the status of various peripherals and calls functions to service those peripherals when they need it (or when an interrupt occurs). You want to know how much time the code spends servicing the peripherals.
One way to do this is to set the output bit at the start of the idle loop and clear it at the end. The time that the pin is high is the amount of time it took to execute the entire loop, including calls to service peripherals and interrupts. On a real oscilloscope trace, you would see a fixed start point (assuming you trigger on the rising edge of the bit) but the end point would be a band. The range of values indicates the variation in the amount of time to get through the idle loop. Figure 3 illustrates the maximum and minimum for such an oscilloscope trace.
Here is an oscilloscope trace of an idle loop showing maximum and minimum execution time. This indicates the range of execution times needed to service all the peripherals.
You can do something similar for a specific function. Say you have a complicated interrupt or sensor processing function that has a varying execution time. You could set the output bit at the start of the function and clear it at the end. This would give you a range of processing times, similar to the idle loop concept.
A SIMPLE LED
Sometimes you just want to know if an event occurs. Maybe early in the debugging process, you just want to know if data is being received on the USB or serial port, or if a particular point in the code is reached. You can, of course, just light an LED. But what if the condition is transient, too short to reliably see on the LED?
What I often do in a case like that is turn the LED into a one-shot. A hardware one-shot IC (monostable multivibrator) generates an output pulse of fixed duration regardless of the length of the input pulse. What this means for an LED is that when the trigger event occurs, the LED will always light for, say, 250 ms, regardless of the duration of the triggering event. This makes any event visible.
The way I typically implement this in firmware is with a counter that is set to some value when the trigger occurs and the LED is turned on. Each time the timer tick occurs, the counter is decremented if it is nonzero. When it reaches zero, the LED is turned off.
As an example, say I want to know when a specific message is received on the serial port, or I want to know when a specific function is executed. Say my system uses a timer tick that occurs 200 times per second. When the desired event occurs, I would call a function that turns on the LED and sets a counter to 50 (i.e., 0.25 × 200). Each time the tick occurs, the counter is decremented unless it is zero. When it reaches zero, the LED is turned off.
— ADVERTISMENT—
—Advertise Here—
This approach has the side effect of making the one-shot retriggerable. If the message is received continuously or the function is invoked rapidly, then the LED will stay on. I have found this to be very useful in some situations.
FINDING SPARE PINS
Sometimes you are using every pin on the device and finding pins for debug outputs isn’t easy. Let’s review some ideas for “borrowing” pins that ordinarily have other uses.
Shunt Jumper or Pushbutton Switch Read by the Microcontroller: Connect it to the microcontroller through a resistor, say, about 1 kΩ. After reading the value of the jumper or switch, make the pin an output and send debug data to it. If you need to read the pin continuously in the code, you will constantly be switching it from input to output and back. The resistor is used to prevent the pin from being damaged if it is driving into a shorted shunt jumper or a closed switch. You will want to size the resistor to ensure that you don’t exceed the output current capability of the pin.
LED Drive Pin: Say you have a heartbeat LED that blinks once per second. Every timer tick, send a serial debug stream to it, ending in the correct on/off state. The debug bits will be too fast to be visible on the LED, but you can see them with an oscilloscope. You will have to edge trigger the oscilloscope.
Use a Pin When It Doesn’t Matter: For example, the clock on many SPI devices is ignored when the chip select (CS) pin is inactive. So you can transmit a serial debug stream on the clock pin when the device is not being used. You have to be careful not to interrupt a normal transmission to the device, or to any other devices that are sharing the SPI bus. You might need an external gate to insure that the oscilloscope doesn’t trigger on a normal SPI transmission. You would use an AND gate connected to the clock pin and the CS pin so that the output is held low when the CS pin is low.
I2C or SPI UART: If you want to use a UART but there isn’t one available, there are I2C and SPI UARTs that you can connect to an I2C or SPI on the microcontroller. The NXP Semiconductors SC16IS741AIPWJ is one such device. You could do something similar to the external RS-232 interface mentioned earlier and put an I2C/SPI UART on an external board, using a header on the target board to connect it. Of course, this uses up at least two pins for an I2C interface and four for an SPI interface. And if you don’t have a spare I2C or SPI interface on the microcontroller, you have to implement the protocol in firmware. This has the potential for significant performance issues and may defeat the purpose of using this approach. However, as a tool for postmortem capture of the internal state of the microcontroller, this is a very viable approach if all the microcontroller serial ports are already used.
USING MCU MEMORY
If all you have is a single pin that you have to use as a serial port by implementing the functionality in firmware, you can still get some real-time information without too much performance impact. You create an array of bytes (just an unsigned character array in C) that you manage as a circular buffer. You write the state values or the 8-bit status bytes to the array. When you hit the breakpoint or error condition in your code, you dump the array to the serial port, using the write index to the array to indicate where the oldest byte is so that you can send the bytes in order. This gives you a history of what happened prior to the event, the same as if you were sending the information to a serial port in real time. The catch is that you are limited to the amount of memory you can spare, which may not reach back in time far enough to see the cause. You may have to be very judicious about what you store in that array to make the best use of it. The C code for writing to the array would look something like Listing 2.
Listing 2
Code for writing debug history array. Send to host as part of post-mortem debug.
Void WriteArray(ByteToWrite)
{
// DebugIndex and DebugArray are defined and initialized
// externally and globally.
DebugArray[DebugIndex] = ByteToWrite ;
DebugIndex ++ ;
If (sizeof(DebugArray) == DebugIndex) DebugIndex = 0 ;
}EASE OF DESIGN
Sometimes you have to be creative to see what is happening inside that piece of high-tech plastic. Hopefully, I’ve given you some ideas that will make debugging your designs a little easier. 
SOURCES
MAX3232 RS-232 driver IC
Maxim Integrated | www.maximintegrated.com
SC16IS741AIPWJ Single UART
NXP Semiconductors | www.nxp.com
PUBLISHED IN CIRCUIT CELLAR MAGAZINE • JULY 2016 #312 – Get a PDF of the issue
Sponsor this Article
Stuart Ball recently retired from a 40+ year career as an electrical engineer and engineering manager. His most recent position was as a Principal Engineer at Seagate Technologies.
