Dumping parallel NAND flash devices is more difficult than working with simple SPI memories, requiring either expensive tools or a ‘hacked-together’ solution. But a new tool called the Glasgow Interface Explorer makes this and many other hardware hacking tasks easier than ever.
In my January 2024 article in this column, I described several ways of looking at “binaries”, a term often used for images of memory extracted by attackers. It can be helpful to analyze them for left-over secret data that might have been copied into the system, or to confirm features such as encryption have been correctly enabled. In that prior column, one item remained on my “to-do” list: figure out a way to extract a binary out of a parallel flash memory chip. In this column, I’m going to finally finish the job.
As I described in the previous issue, I removed the flash chip from the target PCB, and soldered it upside down onto a carrier board. If the target device was one that had existing drivers for tools such as OpenOCD I could have also just used the JTAG debug interface to read the data out. But removing the chip from the board is the “guaranteed to work” solution, so I went with that after others failed.
Figure 1 shows the flash memory chip mounted upside down on my test board, as I described in the prior issue. In the background is a new tool called the Glasgow Interface Explorer, or Glasgow for short. Let’s look at what is inside Glasgow (the tool, not the city).
My parallel NAND chip (front) is connected to the Glasgow interface explorer (back).
Introducing Glasgow
At first glance, Glasgow, shown without a case in Figure 2, might seem to be yet another hardware hacking tool. Lots of tools are used for interfacing to our digital word, including logic analyzer, USB-serial adapters, and more.
The internals of the Glasgow PCB.
What makes Glasgow so interesting is the architecture shown in Figure 3. Glasgow is based on a small (iCE40) FPGA. There are also lots of FPGA-based hardware hacking tools, my own ChipWhisperer has an FPGA at its heart as well for example. But Glasgow’s magic comes from an open-source synthesis toolchain. It takes FPGA logic defined in Python, and in a matter of seconds synthesises it into a new bitstream. This works cross-platform on Windows, Mac, and Linux.
The Glasgow architecture uses an iCE40 FPGA.
The amazing thing here is that you only implement in hardware the exact features you require at the time you are using the tool. Are you changing an input and output pin mapping? Rather than sending a command to some more complex logic in the FPGA, Glasgow enables you to simply synthesize a new FPGA design with the pin mapping you want.
And because both the host tool and FPGA definition are written in Python, it makes it very easy to work with custom protocols. While writing this article, I tweaked some features of the parallel FLASH read algorithm, and found complex modifications, such as changing the state machine path, was much easier with Python than doing it in Verilog.
On top of the special architecture, the actual interface side has handy features you’ll use in real life. There are two banks which can be set to separate voltages, and you can either use Glasgow to generate a voltage or take a reference from your target system. In my case, this is handy since I need to power my NAND flash chip at 1.8V. By using Glasgow, I didn’t need an additional external power supply.
I’ll leave my introduction to Glasgow there, as we need to get back to our flash chip and see how Glasgow can help us read this device.
Glasgow and ONFI
The specific Glasgow “applet” we are using is called memory-onfi . In this case ONFI means Open NAND Flash Interface (ONFI), which is a group that defines standards for talking to parallel FLASH devices. It defines a series of strings that give us useful information about the NAND flash device, such as layout and timing requirements.
In my case the NAND flash datasheet didn’t even mention ONFI support, so it may not be obvious. But after digging into the parameter rows in the FLASH chip, I discovered one of the rows included the characters “ONFI”, which means it was returning a standard set of parameters that the memory-onfi applet in Glasgow can parse.
For basic memory reads we only need a small subset of the information, enough that we could simply read it from the device datasheet. But the memory-onfi applet attempts to read this data from the flash memory itself, which turns out to be a good way to check if the connection to the flash memory is stable. If things are working, the results will be like Listing 1, which is the command I ran as well as the output of the applet.
Listing 1
A partial output of the identify command provides lots of information about our flash memory chip.
$ glasgow -v run memory-onfi -V 1.8 identify...I: glasgow.applet.memory.onfi: port(s) A, B voltage set to 1.8 VI: glasgow.applet.memory.onfi: JEDEC manufacturer 0x2c (Micron Technology) device 0xacI: glasgow.applet.memory.onfi: ID signature: 2c ac 90 15I: glasgow.applet.memory.onfi: ONFI revision 1.0I: glasgow.applet.memory.onfi: ONFI manufacturer information:I: glasgow.applet.memory.onfi: JEDEC ID: 0x2c (Micron Technology)I: glasgow.applet.memory.onfi: manufacturer: MICRONI: glasgow.applet.memory.onfi: model: MT29F4G08ABBDA3WI: glasgow.applet.memory.onfi: date code: (not specified)...I: glasgow.applet.memory.onfi: Memory organization:I: glasgow.applet.memory.onfi: page size: 2048 + 64 bytesI: glasgow.applet.memory.onfi: partial page size: 512 + 16 bytesI: glasgow.applet.memory.onfi: block size: 64 pagesI: glasgow.applet.memory.onfi: LUN size: 4096 blocks; 262144 pagesI: glasgow.applet.memory.onfi: target size: 1 LUNs; 4096 blocks; 262144 pagesI: glasgow.applet.memory.onfi: address cycles: 3 row, 2 columnI: glasgow.applet.memory.onfi: bits per cell: 1I: glasgow.applet.memory.onfi: bad blocks per LUN: 80 (maximum)I: glasgow.applet.memory.onfi: block endurance: 100000 cycles (maximum)I: glasgow.applet.memory.onfi: guaranteed blocks: 1 (at target beginning)I: glasgow.applet.memory.onfi: guaranteed block endurance: 0 cyclesI: glasgow.applet.memory.onfi: programs per page: 4 (maximum)I: glasgow.applet.memory.onfi: ECC correctability: 4 bits (maximum, per 512 bytes)I: glasgow.applet.memory.onfi: Electrical parameters:I: glasgow.applet.memory.onfi: I/O pin capacitance: 10 pFI: glasgow.applet.memory.onfi: timing modes: 0, 1, 2, 3, 4I: glasgow.applet.memory.onfi: program cache timing modes: 0, 1, 2, 3, 4I: glasgow.applet.memory.onfi: page program time: 600 us (maximum)I: glasgow.applet.memory.onfi: block erase time: 3000 us (maximum)I: glasgow.applet.memory.onfi: page read time: 25 us (maximum)I: glasgow.applet.memory.onfi: change column setup time: 100 us (minimum)Listing 2
Reading a section of pages is a simple reliability check, but repeatable errors won’t be caught.
glasgow run memory-onfi -V 1.8 read 0 1000 test001.binglasgow run memory-onfi -V 1.8 read 0 1000 test002.binglasgow run memory-onfi -V 1.8 read 0 1000 test003.binglasgow run memory-onfi -V 1.8 read 0 1000 test004.binListing 3
Modifying the host software is a simple Python change, no need to recompile anything. Here I added a simple reliability hack.
while count > 0: self.logger.info(“reading page (row) %d”, row) chunk = await onfi_iface.read(column=0, row=row, length=page_size + spare_size) chunk1 = await onfi_iface.read(column=0, row=row, length=page_size + spare_size) chunk2 = await onfi_iface.read(column=0, row=row, length=page_size + spare_size) chunk3 = await onfi_iface.read(column=0, row=row, length=page_size + spare_size) if chunk == chunk1 == chunk2 == chunk3: pass else: self.logger.info(“oops, multi-read failed, retrying...”) continueHow stable are these reads, you might ask? Figure 1 shows a BGA device mounted with long leads and only a far-away decoupling capacitor. That would certainly prompt some questions. My first attempts at reading the data weren’t successful, as I received errors about the CRC on the configuration pages. If you see the companion GITHub page or my blog post linked in the resources, you can see the modifications I used to print the full errors. The important summary of the errors was it appeared that the Glasgow was “skipping” bytes, that is there were bytes missing in the datastream that I expected.
My guess was something was causing the device to clock additional read cycles, as it would automatically increment the read address on every edge of the read pin. Based on my poor breakout quality, bad Signal Integrity (SI) was my first guess.
Look at the signals on the FLASH pins, which look something like in Figure 4. The ringing on the pins was causing additional read cycles to be triggered. The errors were more likely on specific sequences such as long strings of zeros, which indicates bad SI because the lines were loaded in a specific sequence.
Measured at the pins of the NAND chip, high levels of ringing can be seen.
To solve this problem, I added additional series resistors on a few of the pins such as the read enable pin. The Glasgow already has some series resistance but by adding a little more, I improved results considerably.
Fix it in Software
Before going too far, it’s a good idea to check a few dumps of the same data. Using the commands from Listing 2, I dumped the first 1,000 pages to compare. This still showed occasional errors. I could either fix the hardware or add some improvements into the software.
I should emphasize these “improvements” are major hacks, but they worked for me. The general idea is shown in Listing 3 (See the companion GITHub repository for the full code). But the idea is that it attempts to read the same block multiple times, and if it’s not the same it will retry again.
The part of this that remains a major hack is that the errors were not random, but instead tended to occur with specific sequences of bytes read and tended to still be just a dropped byte. If the same byte was dropped in all 4 reads, it will incorrectly assume the data is good.
In the future I may try this with a more reliable PCB to confirm if SI was really the issue, or if were timing issues.
Exploring a Digital World
This was just one example of what you can do with Glasgow. The tool has had a huge amount of work put in by the primary maintainer (‘Whitequark’), and once I started digging into the implementation, I was amazed to find that this tool rebuilds an entire FPGA image for every small change, but it feels like you’re using any simple software/firmware interface tool.
Glasgow has all sorts of useful features like automatic baud rate following of serial ports, as well as more esoteric protocols such as parallel floppy drive interfaces, features that simply don’t exist in other tools It saved me from a major “to-do” I had left in my previous article. I owe a debt of gratitude to Whitequark. It saved me from having to implement my own parallel NAND interface, which was the original plan detailed in the previous article.
RESOURCES
Copies of code listings, including more complete context in the repository at https://github.com/colinoflynn/circuitcellar-EmbeddedSystemEssentials
Glasgow interface explorer homepage: https://glasgow-embedded.org/latest/intro.html
PUBLISHED IN CIRCUIT CELLAR MAGAZINE • JULY 2024 #408 – Get a PDF of the issue
Sponsor this Article
Colin O’Flynn has been building and breaking electronic devices for many years. He is an assistant professor at Dalhousie University, and also CTO of NewAE Technology both based in Halifax, NS, Canada. Some of his work is posted on his website (see link above).
