MCU Real-Time Tracing

How to Trace MCU Code in (Almost) Real Time

How do you trace code execution in an MCU? This month, Stuart looks at the problem and offers real-time solutions.

One Saturday, three software engineers and myself were “volunteered” to come in early to work on this. We didn’t make much progress, but we were using a logic analyzer that just wasn’t up to the job. By the end of the day, the project manager came down to the lab and asked what was needed. I said “Get me an HP1630 logic analyzer and I can solve this.” Kind of a brash statement, I know, but I had used the HP1630 at a previous job and knew its capabilities.

The manager went back to his office and rented an HP1630 for expedited delivery the next week. The analyzer arrived on Tuesday morning and by Tuesday afternoon, I had identified the problem—it was a subtle logic error in the hardware design on one of the boards. My next assignment was to get quotes and put in a purchase requisition for two HP1631 logic analyzers. Once they arrived, nobody wanted to use the old one. I don’t even remember what brand it was.

Some years later, at the same company, I was working on a problem in a paper handling machine where an error in some MCU code caused a spacing issue that couldn’t be detected until the paper had traveled a few feet down the track. I was sending tracepoints, but in that case, the HP1631 didn’t have enough memory to look back far enough and I used our new HP1651. Even the tool that had saved the day a few years earlier had to be replaced by one that had the right features for this problem. The point is that sometimes having the right tool makes all the difference.

MCU TRACING

When working with an MCU, you can’t connect a logic analyzer to internal memory and data signals. Debugging an MCU often involves a trade-off between pins and software. The more pins you dedicate to debugging, the fewer you have for whatever the MCU is supposed to be doing. And the more you have to change the software to support debugging, the more you change how it works and the more likely it is you will mask a problem. As I mentioned in my article about JTAG debugging (Circuit Cellar 427), a JTAG debugger takes four pins of the MCU. If you are using an ARM MCU, it’s at least two pins, three if you want the trace output. Depending on the debug capabilities of the JTAG debugger and the MCU, you may be able to monitor some variables while the MCU is running, but the speed is limited; it’s not truly real-time. A JTAG message requires sending multiple bytes to the hardware registers, limiting the speed. You can send debug messages to a serial port, which is one pin, but everything stops while sending the message.

Any code you add to send trace information is limited by the speed of the hardware and the amount of information you send. Sending a string such as “Tick ISR start: 12:23:44” takes 2.9ms to send over a 115200 baud serial port. It requires modifying the code to add a function to send the message, and if the serial port UART is already full sending a previous message, the message doesn’t get sent until the UART has space.

Ideally, you would have tracing capability that is real-time, requires no additional code that can affect the timing, and has instantaneous transfer so there are no delays. Don’t hold your breath. In the real world, some hardware is needed and some software changes are needed to get the information out of the MCU “black box.” Even MCUs that have a real-time trace buffer, such as some PIC32 variants, have limited buffer size; only so much memory can be dedicated to this inside the MPU itself.

A SINGLE PIN SOLUTION:

As mentioned, some ARM processors have a single wire trace capability that will allow tracing information to be sent. However, it requires that the ARM Serial Wire Debug (SWD) be enabled, which uses three pins. The concept here is similar but applicable to any CPU with an available serial port. The general idea is that you will send trace data as single bytes. Each value represents a tracepoint in the code. For example, value 00 may indicate code start, 01 may indicate when a tick ISR is entered, 02 might indicate a specific detected error. In theory, you could have 256 codes, but in practice you may want to limit it to printable ASCII code values, which gives you 95 codes. The goal of this method isn’t to send message strings, but to send tracepoint information that can be expanded to whatever message string you want at the receiving end.

Speed—The almost-real-time nature of this solution depends on three things; an available asynchronous serial port output, single-byte code transmission and high baud rates. A Microchip ATMega324 MCU operating at 18.432MHz can achieve a baud rate of up to 1.15MBaud, or 115200 characters per second. A TI TM4C1233D5 can achieve baud rates of 1MBaud, or 100,000 characters per second.

USB—The other side of this solution is a serial TTL-to-USB converter, as shown in Figure 1. This was an inexpensive unit that I purchased online. FTDI makes a module that is also inexpensive (under $25) and would work as well. The one I used has a 6-pin header; FTDI makes a few versions, including one with a 3.5mm audio connector. Probably any USB to TTL/CMOS serial converter would work as long as the output voltage matches the target board (5V, 3.3V, etc.) and the module handles transmission speeds as fast as your MCU can send the data.

FIFO—Although using a serial port in this way runs the risk of overflow, many MCUs have built-in FIFO buffers on the UARTs. So if the FIFO is 8 bytes deep, you can safely write 8 bytes before a byte is lost. You can even write a few bytes back-to-back, as long as the average rate doesn’t exceed the transmission rate.

AN EXAMPLE

Figure 2 shows the schematic of a circuit to demonstrate this tracing method, based on the Texas Instruments (TI) TM4C1233D5 MCU. Connector J2 is for programming the MCU, connector J4 connects to the USB-to-serial module. D1 is a heartbeat LED that blinks once per second, D2 blinks once each time the Test button is pressed. Debug information is transmitted on UART 1, which is on pin PB1. As is typical with these examples, the schematic is part of a larger development board; I deleted everything that wasn’t relevant to this article. Code was written using the TI Code Composer Studio suite and the TI Tivaware libraries.

The TM4C1233D5 UARTs have 16 byte FIFOs, plenty of depth for anything happening in this example. The software has a timer tick interrupt using the SYSTICK timer that operates at 1000Hz. For this demo, I send code ‘w’ when execution starts after reset, code ‘A’ for the timer tick interrupt, ‘B’ for the 1 second blink, and ‘C’ when the Test button is pressed.

In the example code, writing to the UART is a blind write using a HWREG statement. There is no checking for whether the FIFO is full, so writing to a full FIFO will cause data loss. Ordinarily writing to a UART would be preceded by a check for available FIFO space, but that requires additional code and makes the tracing more intrusive.

My philosophy on this approach is that if you are sending faster than the UART can accept data, you need to be tracing fewer items. It’s always possible to do a normal Tivaware call to the blocking UART write, which will block until the UART has space. But that impacts timing and can potentially affect whether things work correctly. My goal here is to make the debug output as unintrusive as possible even if I have to break a rule to do so.

To generate the test outputs, I used a simple Python program that just reads the codes from the USB comm port and stores them in a text file. The capture program runs for 5 seconds, although it will also stop if an end-of-capture tracepoint is sent. In a real application, you would probably take out the 5-second timeout and just let the Python program run until the ‘x’ end-of-capture code is received. After capture stops, the program converts the codes to text in another file. The Python program is included on the Circuit Cellar website. Figure 3 shows a snippet of the raw data from the serial port capture. The raw ASCII data is written to an output file.

After the data is captured, the program opens the output file and parses the byte codes to their specific meaning, as shown in Figure 4. Obviously the byte code translations seen here are unique to this example application and you would have different translations in your application. The program writes the translated values to another file so you can examine it later. I use the same file names over and over, but if you wanted to modify the Python program for your needs, you might want to add a timestamp to the file name so you aren’t overwriting the files each time. There are less clunky ways to do this, I just wanted to demonstrate the principles.

Reserved Codes—MCU software and the Python code reserve four codes for special use. ‘w’ is a start code that triggers the Python program to start capturing. It allows the Python program to be started before the MCU is running, such as when downloading new code. ‘x’ is a stop code that stops capture and is intended to be sent as essentially a breakpoint. ‘y’ is a start-of-value code that initiates capture of a value in hex ASCII format. ‘z’ stops the hex ASCII capture.

So the MCU can send a hex ASCII value for display. Using the ‘y’ and ‘z’ codes does affect the intrusiveness of the debugger since it requires sending at least four extra characters (‘y’, two characters for a byte value, ‘z’). The longer the variable you need to send, the more bytes are needed. To send the hex value F123, the MCU would send yF123z. This can be used as a way to capture one or more variables after a breakpoint; when the breakpoint is reached, the variables are sent using ‘y’ and ‘z’, and then ‘x’ is sent to end capture. Used in this way, impact of performance is minimal because variables are sent after the breakpoint (really tracepoint) is reached and sending as many variables as needed doesn’t impact anything since the capture is going to stop immediately after. But if sending more than one variable, you’ll want to use a function that waits for the UART to be ready so there is no overflow, as shown in the MCU code.

To demonstrate this, I incremented a tick counter each time the tick ISR occurs, and then sent the value followed by a terminating ‘x’ when the test button is pressed. Figure 5 shows a snippet of the captured trace where the value of the tick counter is displayed when capture stops. In this example, the tick counter is just to demonstrate the concept and will be different every time you hit the test button. You could go further and, knowing what is transmitted, label each of the variables in the Python code. I didn’t do that in the example Python program. It might be a bit tedious since every time you change it, you have to change the Python code (or some text file that defines the variables you’re sending).

There is nothing to prevent you from defining another reserved code (say ‘u’ and ‘v’) as indicators of names for the tracepoints. The MCU code would send the names with the variable values. Obviously the MCU code would have to be modified to send the names and the Python code would have to recognize the reserved codes. Since all this is sent after a breakpoint, it won’t affect code execution. I didn’t implement this and I can’t think of too many situations where it would be needed.

You could use this as a more general debugger when there aren’t enough free pins for JTAG or another debugger. You wouldn’t even need the tracepoint outputs, except when you hit the breakpoint and then transmit all the relevant variables followed by the stop character. You don’t even need to stop the code, which could be an advantage when debugging something that will break if the code just stops.

GOING FURTHER

SW FIFO—The 16-byte FIFO buffer in the TM4C1233D5 UART hardware is certainly adequate for this example. But in a MCU with no FIFO, you might want to add a FIFO in software. The FIFO won’t magically allow transmission to happen faster than the UART can send, but a software FIFO can average out brief bursts of characters. This is going to be more intrusive because instead of sending directly to the UART, you will be calling a function that adds a byte to the buffer, then using an interrupt or polling to determine when the UART is able to accept more data. And the FIFO has to be large enough to handle the worst-case number of data bytes that need to be sent.

Two-Way Communication—If you are willing to use both the transmit and receive ports of the MCU, you could use this tool as a more sophisticated debugger. When a breakpoint (specific tracepoint) is reached, the MCU code would stop and go into a simple-minded monitor program that accepts simple commands via the serial input. The command “M xxxx” might display 64 bytes of memory at location xxxx. Unlike a dedicated debugger, it wouldn’t be aware of the locations of specific C variables, so it’s a little bit clunky. This would require a more sophisticated GUI-based software tool, such as PuTTY or HyperTerminal. In general, though, I see the ability to send specific variables at end-of-capture to be more useful.

Limitations—This solution isn’t intended to be a complete debugging tool. What does do is let you capture ongoing code execution so you can trace what the code is doing as it runs, and to stop at a specific point in the code and display the result of selected variables. So it does have some limitations.

Integration—The most obvious issue is this tool isn’t integrated into existing debugging tools. You can kind of integrate it in the sense that a debugger breakpoint will stop the trace output, so you can look at the trace history once it times out. But you still need an external program to capture and parse the serial port output. And you can’t just click on a variable to see the value like you can in an integrated debugger. Lack of integration into tools is probably the biggest limitation of this technique. But if you need a long list of tracepoints like that problem I solved with the 1651 logic analyzer, this tool gets you closer to real time tracing than normal debug tools do.

Limited Tracing—This tool provides real time tracing, but it can’t trace individual machine code instructions or C statements. It is limited to where you put debug outputs in your code. It can’t pause the code at a breakpoint, although you could stop with a tracepoint the same as you would a breakpoint. But you won’t have the amount of debug data available that you would with a debugger.

External Software—This tool only works if you have some software on your PC to capture the serial port output. The provided Python program works, but you may want something more sophisticated. You could use off-the-shelf software such as PuTTY but you want something that can log the trace codes to a file and then you will want to parse them. If you are only tracing a few codes, parsing may not be needed.

Software Update—Since the trace outputs are embedded in the code, you have to recompile and reload the code any time you add or remove a tracepoint.

Timestamps—I originally had the Python code capture the timestamp of each character. But since the USB driver packetizes the data, the timestamps aren’t very reliable. I took that code out. If you wanted to use timestamps you can add it back in, although the accuracy is on the order of milliseconds, not microseconds. The original line of code is noted at the end of the Python file in a comment if you want to try it. Adding line numbers to the parsed data would probably be more useful. But, the Python program is a proof-of-concept, not a finished product.

Speed—In theory, the speed is limited by the serial port speed of the USB converter. But in practice, it is limited to the maximum baud rate the Windows VCP (virtual com port) driver supports. The module I used, based on the CH340 chip, will work to 921600 baud, but the example code isn’t sending continuously at that speed. USB transfers are not interrupt-driven, but polled, with a 1ms sample interval. So while 921600 baud will work, there is no guarantee it will be a sustained data rate. The CH340 can support 921600, as can the FTDI FT232R. The maximum sustained USB rate (different from maximum programmable baud rate) is a function of the VCP driver and how stressed the OS is in handling other programs.

As a test, I modified the MPU code to disable the tracepoint outputs and instead sent the characters A-Z at 921600 baud, 200 times (total of 56ms). I modified the Python program to look for that pattern and report any mismatches; the serial adapter, PC, and Python program were able to keep up. I also tried it on a serial module that uses the FTDI FT232RL; it worked as well. I didn’t test on Linux, but I’d expect it to work, assuming the Linux VCP driver for the module supports the same capabilities. The MCU code changes are still in the files on the Circuit Cellar website, just commented out.

Single-Pin Visibility

There are no magic methods to do real-time tracing in an MCU like we did in the old days with hardware emulators and logic analyzers. But this technique lets you have a reasonable degree of program visibility using a single pin and minimum impact on the code timing. It is useful in those cases where you need the history of the execution steps to see where something went wrong.

RESOURCES
ARM | www.arm.com
Microchip | www.microchip.com
Texas Instruments | www.TI.com

Code and Supporting Files

PUBLISHED IN CIRCUIT CELLAR MAGAZINE • APRIL 2026 #429 – Get a PDF of the issue

Sponsor this Article