Build a SoundFont MIDI Synthesizer (Part 2)

The Firmware and Circuitry

In Part 1, Brian focused on the SoundFont standard. This time, in Part 2, he discusses the Teensy 4 firmware that implements a MIDI wavetable synthesizer. He also details the circuitry of the project—which is quite simple thanks to the Teensy 4 MCU doing much of the heavy lifting.

In this article, I’ll describe the Teensy 4 firmware that I wrote, which implements a MIDI Wavetable Synthesizer. Then I’ll cover the schematic diagram of the project, which is quite simple, since the Teensy 4’s MCU handles most of the functionality itself.

TEENSY 4 FIRMWARE
The Teensy 4 is the latest member of the Teensy MCU module family. It uses NXP Semiconductors’ iMXRT1062 MCU, which runs at 600MHz and contains 2MB of program flash memory and 1MB of SRAM. Unlike the earlier Teensy 3.6 module, the Teensy 4 module uses a smaller PCB in a DIP28 format. This keeps the price of the module low ($20), although I preferred the greater I/O capability of the Teensy 3.6. That said, for this project, there were sufficient I/O pins to handle everything. Figure 1 shows the Teensy 4 module. While not visible in the image, the underside of the PCB contains footprints for more I/O capability, including a footprint for a full-speed (SDIO) SD card interface.

The firmware running on the Teensy 4 consists of three main components:

1) The AudioSynthWavetable object, which is part of the Teensy Audio library. This was written by a group of Portland University students. I modified it for this project, as mentioned in Part 1.
2) The MIDI note handling engine, which handles incoming MIDI messages. It parses the Note On/Off messages, and assigns each playing note to a dedicated AudioSynthWavetable object. MIDI control messages for Modulation (“Mod” wheel) and Channel Aftertouch are routed to the vibrato function. MIDI volume messages adjust the gain of the output mixer. Seven other midi controller slider messages are mapped (using the “Learn” Menu) to adjust the 7-band Graphic EQ filters.
3) Seven-band Graphic EQ. I implemented this totally in software, using the Audio library’s BiQuad filter objects (Figure 2) All filters are 4-stage biquad filters configured as bandpass filters. I set the Q values for each filter to allow that filter to provide an adequate response up or down to the adjacent filters. All the center frequency and Q values can be modified, if desired, by changing values in the setFilters() routine (at the end of the program listing).

Now, let’s look at each program section in more detail. The AudioSynthWavetable object is a complex bit of code. Apart from some parameter-initializing routines, its main function is handled—as are all Audio library objects—in the update member of the class. The update routine gets called automatically at the 44,100Hz sample rate. It is responsible for stepping through the selected voice’s wavetable at a rate that will produce the desired note frequency. This is done basically by using the DDS (Direct Digital Synthesis) principle. For any given voice, there will be several sample waveform tables—one for each “region” of the keyboard. So, depending upon the frequency of the note being played, a pointer to the correct waveform array must be determined.

As mentioned in Part 1, when the MIDI keyboard sends a Note-On message, the sound synthesis process must break down the waveform generation into two parts: the initial “attack” portion, which consists of the first “x” samples in the wavetable array, and the “loop” section, which consists of the rest of the wavetable array. After taking samples from the beginning of the array (the attack), for a period of time defined by that voice’s attack-time parameter, the wave generation is pointed to the loop section of the array. It then loops through this section of the array continuously, until a Note-Off message is received from the keyboard controller. Then it enters the “release” phase, in which it repeatedly scans the loop section of the waveform array, but gradually introduces attenuation to the waveform to allow it to fade out exponentially. This would occur with an acoustic musical instrument played in an actual room (with reverberation). A new Teensy 4.1 module came out since I wrote this article. This module uses the larger Teensy 3.6 footprint, and contains more memory, I/O lines, a built in SD card socket and an Ethernet port.

Modulation effects such as vibrato, which are specified in certain sections of the SoundFont file, must also be handled by the update routine. While the original AudioSynthWavetable object handled any vibrato that was specified in the SoundFont file, it did not allow for vibrato to be introduced by the MIDI keyboard’s Modulation controller (Mod wheel) or Channel Aftertouch. I added that feature in the main section of my program, in conjunction with a slightly modified AudioSynthWavetable library object.

MIDI NOTE HANDLING
The MIDI note-handling engine is performed using the MIDI class itself. The MIDI class is set up to handle incoming MIDI messages on an interrupt-driven basis. So, whenever a MIDI message comes in, a background routine picks up that message and parses it into one of many different MIDI message types. My program basically recognizes the following MIDI messages:

1) Note On/Off
2) Continuous Controller: MIDI Volume, Mod wheel and EQ settings are recognized
3) Program Change: selects the different voices available on the SD card
4) Channel Aftertouch: if you press down hard on (any) key, the Channel Aftertouch message will be sent. This program responds by introducing vibrato at an amplitude dependent upon the key pressure exerted.

We must define callback routines for each of the above types of messages. In the case of Note On/Off messages, a note allocation routine must be performed. Basically, every Note-On message results in assigning one of 48 discrete AudioSynthWaveform objects to that particular note. When a Note-Off message arrives, the program must scan through all 48 AudioSynthWaveform objects to determine which one of them is “playing” that particular note. Then, that particular object is sent a “Stop playing” command.

But it’s a bit more complicated than that. Even after you release a key corresponding to a particular note, the wavetable object that is playing that note is not yet finished with it. It has merely gone into the release phase, where it is still sounding the note, but at a decaying amplitude. The amount of this release time is entirely dependent on the release-time parameter in the SoundFont file. Additionally, that release time can vary, depending upon the frequency of the note being played (that is, what region of the keyboard has triggered the playing of that particular note). Therefore, you must call a function—isPlaying()—to determine whether a particular wavetable object is still playing a note.

The routine freevoices() must be called repeatedly. It looks at each active wavetable object, using isPlaying(), and once it returns a false condition, that particular wavetable object can be returned to the idle (free) group. Once freed up, it can be re-assigned, as necessary, for future incoming notes. Because of this release time, many voices can be sounding at any given time, even though you only have 10 fingers with which to play the keyboard. Granted, many sounds will be virtually inaudible at the end of the release. This explains why I allocate 48 wavetable objects to handle so many sounding voices.

The Control Change message for Volume is easily handled by adjusting the gain of the two inputs of the final mixer object. This mixer combines all seven signals from the 7-band graphic EQ. The Control Change message for the Mod wheel is more complicated. I must basically take the Mod wheel value (0-127) and store it in the following two variables (after dividing it by 1,024):

sd0[x].MODULATION_PITCH_COEFFICIENT_INITIAL
sd0[x].MODULATION_PITCH_COEFFICIENT_SECOND

where x goes from 0 to the number of keyboard regions minus 1. This is necessary so that notes that fall into all the various regions (based on frequency), will each respond to the Mod wheel (vibrato) messages.

PROGRAM CHANGE
The last type of controller message is the Program Change message. This takes the form of a number in the 0-127 range, which correspond to various “programs” (voices). When these messages are received, the program attempts to load in a SoundFont file from the SD card, with a primary filename from “1” to “127.” Two files are loaded: an xxx.h file and an xxx.cpp file (where xxx is in the 0-127 range). These extensions were defined in the original Python program, written by the developers of the AudioSynthWaveform object. As mentioned in Part 1, that Python program converted standard SoundFont files into two files with .cpp and .h extensions that were meant to be “included” in the synthesizer program itself. The original developer’s firmware could play only the one voice that was included in the program code, itself. Even though I have changed both the Python program and the Teensy 4 firmware to enable loading user-selected voices into SRAM, I retained the original .cpp and .h file-naming convention.

When I first heard of the Teensy 4 module, one of the impressive specs—in addition to the smoking 600MHz clock speed—was the large amount of on-board memory: 2MB of program flash and 1MB of SRAM. A Wavetable synthesizer requires a lot of SRAM to handle complex musical instrument voices. I’ll be honest—commercial wavetable synthesizers contain many megabytes of SRAM, and that’s part of the reason they are so expensive. I felt that the 1MB of SRAM on the Teensy 4 would be enough to handle many useful samples.

It turns out that the Teensy 4’s 1MB of SRAM is not just one big block of memory that is completely free to use for your program’s variables and arrays. (The Stack and Heap take up a small amount of it, of course.) Instead, it is broken up into two 512KB sections. At run time, the MCU copies the program code from flash memory into the lower 512KB section of SRAM memory to boost execution speed. It allocates this program SRAM in discrete 32KB blocks. For this program, that is about 81KB, so it uses up a 96KB block of the lower 512KB block of memory. Allowing for other miscellaneous variables and Stack space, it turns out that only about 380KB of SRAM are available for the sample wavetable, from within this 512KB block.

The upper 512KB block is physically coupled to the CPU in a different fashion than the lower 512KB. It is labeled as DMA memory. It turns out you can use this DMA memory for your sample wavetable, by using the “DMAMEM” directive when declaring that array. However, you can only allocate 512KB for this array. You can’t combine it with whatever is left of the lower 512KB section after the program is transferred there from flash.

So, the maximum sample wavetable is 512KB minus the Heap size. I settled on a sample array of 120,000 elements consisting of 32-bit integers. The sample wavetable array actually stores 16-bit samples, packed two per 32-bit element, so that’s 240,000 discrete samples. Lots of useful voices require sample sizes less than this, but you won’t be able to load in a really good piano voice, as those are quite complex.

The Python program that generates the files used by this synthesizer shows the size (in bytes) of the sample file that you are considering. Each SoundFont file defines the number of regions that were used in producing the file. Each region covers a range of MIDI notes. If you have chosen a Soundfile for a complex instrument, you may have to pick just a subset of all the available regions, to stay within the 240,000 16-bit sample limit (which the Python would show as 480,000 bytes).

7-BAND GRAPHIC EQ
The last functional block is the 7-band Graphic EQ. I did a lot of experimenting with various filter configurations before settling on the configuration shown in Figure 2. Initially, I thought it would be interesting to try to implement a multi-band parametric EQ using a single, 513-tap FIR filter. In my article “Fancy Filtering with the Teensy 3.6” (Circuit Cellar 346, May 2019 [2]), I described a convolution filter object, which I had designed for the Teensy Audio library, that emulated a 513-tap FIR filter (using FFT/iFFT techniques).

Although that design worked well for some applications, it turned out that trying to force it to handle a multi-band parametric EQ was not particularly successful. The 16- bit, integer-based FIR impulse array did not have the required precision to handle so many different frequency bands. Also, it turns out that a 513-tap FIR filter, using a sampling rate of 44,100Hz, has an FFT (fast Fourier transform) bin size of about 86Hz. The lowest filter frequency that would work at all must be at least 3 times that bin size, or 258Hz. A lot of musical instrument content lies below this frequency, so the FIR filter idea was abandoned.

Instead, I decided to use a bank of seven, 4-stage biquad filters. There is a Teensy Audio library object for the biquad filter. It is easy to use: you just load the object with the type of filter (band-pass in this case), the center frequency and filter “Q,” and it does the rest. Each of the four sections of the filter must be initialized this way. The diagram of this from the Audio Designer application, including the frequencies that I chose for each band, is shown in Figure 2. All seven filters are fed to individual inputs of Mixer1 and Mixer2, and those two mixers are combined in Final Mixer3. You adjust the gain for each of the seven bands by setting the input gains of Mixer1 and 2.

In use, I “map” seven of the MIDI keyboard’s sliders to the seven EQ bands, using the Learn option available when you press the Menu button. This needs to be done just once, when you first power up the synth after programming the Teensy 4’s firmware. After performing that “learn” function, the program saves the MIDI control numbers of the seven sliders you chose to assign to the 7-band EQ. That mapping is stored in the config.txt file on the SD card. Figure 3 shows the bank of sliders on my MIDI keyboard controller. They are nicely suited to setting a Graphic EQ filter.

When playing the synthesizer, moving any or all of those sliders will adjust the frequency response of the 7-band EQ. Also, if you hit the SAVE button, that frequency response curve will be stored in the config.txt file, indexed to the current program number. In other words, the 7-band EQ settings can be uniquely saved for each voice.

Initially, I had decided to save both the keyboard slider mapping and the seven EQ filter gains (×127 voices) to EEPROM, because there were less than 1,000 values to store, and those parameters must be non-volatile. Teensy 4’s iMXRT1062 MCU does not contain any real EEPROM memory. The EEPROM library handles this by using a 64KB block of program flash for EEPROM emulation—spreading your data within this space for wear-leveling purposes. I found that attempting to use this EEPROM library resulted in my program crashing, and, if my memory serves me correctly, I think it also “bricked” the MCU (which is reversable by re-flashing the device using a 15 second press of the Program button on the Teensy 4). On the Teensy Forum [3], I ran into others who had experienced the same problem, so I decided to abandon the EEPROM route and instead store this data to the SD card (in the config.txt file). I expect that this bug has been fixed by now.

There is a Menu option to set the MIDI port on which the synthesizer will respond. If you select “0,” the MIDI OMNI mode will be active. This will respond to MIDI messages on all 16 channels. Otherwise, MIDI channels 1 through 16 can be selected. The LCD display will show what MIDI channel the synthesizer is set to respond to. The MIDI channel selection is non-volatile, and is stored in the config.txt file on the SD card.

As a troubleshooting aid, whenever a key is pressed on the MIDI keyboard, an LED on the synth will turn on, and stay lit until the key is released. If this doesn’t occur, then you have the synthesizer set to the wrong MIDI channel, or there is some hardware issue in the MIDI input port circuitry.

Because I made some changes and enhancements to the AudioSynthWavetable object, it is imperative that you replace the following two original files—synth_wavetable.cpp and synth_wavetable.h—with the same-named files that I provide for this project on the Circuit Cellar article code and files webpage [4].

These two files are located at:

C:\Arduino- 1.8.9\hardware\teensy\avr\libraries\Audio [1]

Note that the initial part of the path shown above should be amended to the folder where your existing Arduino IDE is located, which may be different than on my PC.

I have not tried to run this program on a Teensy 3.6, which would be the only other Teensy module that had anywhere near enough SRAM. The original Audio Synth Wavetable library will work on the Teensy 3.x modules, but must be quite limited by the smaller amount of SRAM available on those modules. To compile programs for the Teensy 4, you must be running Arduino 1.8.9 or newer, and Teensyduino version 1.47 or newer.

CIRCUITRY
The schematic diagram of the unit is shown in Figure 4. There is no power supply shown, because power for the whole unit is supplied by a 5V USB adapter that plugs into the Teensy 4 micro-USB socket. The Teensy 4 module contains a low-dropout, 3.3V regulator that provides power to the Princeton Technology PT8211 Stereo DAC. All the remaining circuitry is powered from the Teensy 4’s 5V pin.

The Teensy 4 module contains a footprint for a flex cable that can be connected to an SD card socket. This is found on the underside of the module. This footprint is connected to the MCU’s SDIO (4-bit) native SD port. I thought it was advantageous to use this high-speed SD card port rather than connect an SD card to the standard SPI port on the MCU for two reasons. First, the wavetable samples can be quite large, and the higher speed of the SDIO port is advantageous (over 4× faster). Second, The Teensy 4 has many fewer I/O pins compared to the earlier Teensy 3.6/3.6 modules. I needed the SPI pins for other purposes.

Choosing to use the SDIO SD card interface was a tough decision, since the SDIO PCB footprint is quite small—the pads use 1mm spacing. I don’t agree with PJRC’s decision to squeeze this module into such a small size, equal to that of the earlier Teensy 3.2 module. If you want to use an SD card with the SDIO port, you need a couple of tiny FPC (flexible printed circuit) sockets, an FPC flex cable and a separate PCB on which to mount the SD card. This adds $10 to the cost, and requires some tricky soldering. Now that the Teensy 4.1 module is available, it would make sense to use it and avoid this external socket issue.

I chose instead to do it my own way. In Figure 5, you can see where I’ve soldered eight wire-wrap wires to the SDIO footprint, and connected them to a small breakout board containing an SD card socket. I applied some black epoxy over the wire bundle, close to the SDIO footprint, to act as a strain relief. The wires are about 4” long—enough to allow me to mount the SD card socket at an accessible point on the enclosure. The SDIO interface is a fast one, so I don’t think it’s a good idea to make the connecting wires any longer than this. I don’t believe it was any harder to solder those eight wires than to solder a tiny FPC socket in place on the Teensy board, and another one at the SD card PCB.

I happened to have an SD card breakout board handy, but it’s easy to solder the wires to a microSD-to-SD card adapter (which comes with most microSD cards). This provides a free microSD card socket without requiring a custom PCB and conventional microSD card socket. While I use color TFT displays for many projects, commercial synthesizer modules generally just use modest LCD alphanumeric displays. The user interface is pretty basic, so I settled on a 20 character × 2-line LCD display. It requires a 6-wire interface to the Teensy’s GPIO lines, and a good Arduino library is available for it.

ARM MCUs
I didn’t have a lot of choice about the type of MIDI port I could use. While earlier Teensy modules supported USB-MIDI (using the on-board micro-USB connector), the Teensy 4 is different. The NXP iMXRT1062 Arm MCU used on the Teensy 4 module is quite different from the Freescale Arm MCUs used on the Teensy 3.x modules. It took some time to develop the multiple USB profiles that were available on the Teensy 3.x modules (Serial, MIDI, Keyboard, Joystick and others). The latest version of the Teensyduino program now contains a USB-MIDI profile, but when I developed this project, that was not available. Therefore, I used a legacy-style MIDI port consisting of a 5-pin DIN socket and a 6N138 opto-coupler. Note that the 6N138 requires 5V of power to run properly, but the open collector output must be returned to the 3.3V supply (via R2), since the MCU GPIO pins are not 5V-tolerant.

When I started this project, I connected the Teensy 4 to PJRC’s audio adaptor board. I had one on hand, and there was some discussion on the Teensy Forum about whether it would work with the much faster NXP iMXRT1062 MCU. I had no trouble getting it to work, but for this project, I only needed a DAC for an analog output signal—not a full codec as is found on the audio adaptor board. Therefore, I replaced the audio adaptor board with a PT8211 16-bit audio DAC, which also interfaces with the NXP iMXRT1062 MCU via its I²S bus. The PT8211 driver, part of PJRC’s extensive Audio Library, uses 4x oversampling (4 × 44,100 =176,400Hz SR) for better sound quality. The PT8211 chip, itself, is a bargain at less than $1, but is not readily available from US distributors. I was able to get a pack of 10 of them from eBay some time ago. However, PJRC sells a tiny breakout board containing the PT8211 and a few passive components for less than $3.

The PT8211’s stereo output is sent to a 3-pole Butterworth low-pass filter made up of passive components and Microchip Technology dual op amp MCP6002. Since most musical instruments—apart from cymbals and the like—don’t produce harmonics beyond 10kHz, the filter is set for about 12kHz. It effectively removes any of the DAC clock signal and high-frequency hiss that can be introduced by limitations in the resolution of the sample wavetables. It also removes noise introduced by all of the 16-bit processing that the MCU must do to implement the 20 software-based mixers needed for the design.

The MCP6002 is a single-rail op-amp capable of rail-to-rail input and output. No negative power supply is needed for this op amp. The SoundFont files are monaural, as is the whole signal generation/processing chain in firmware. You can only get audio DACs in a stereo configuration, and I decided to filter both output channels individually, and provide stereo output sockets. The Teensy 4 is only using 1% to 10% of its available horsepower for this program (depending on how many notes are being played at one time). You could add additional effects like reverb or chorus to the program. In this case, the included stereo hardware may be utilized more fully.

All user input/adjustments are performed using a rotary encoder to choose between various menus and to adjust parameters. Four push buttons are implemented: Menu, Enter, Save and Exit. I am not using Exit currently.

The 20×4 LCD display requires 5V of power to operate properly. I get that from the 5V pin on the Teensy 4 module, sourced from the USB port which, in this project, is connected to a 5V USB power adapter. Even though the LCD display is powered from 5V, its logic pins require only TTL signal levels. Therefore, the 3.3V logic levels coming from the MCU’s GPIO output pins are perfectly fine without any need for level-shifting.

The circuit board containing the circuitry, as well as the external components before mounting in an enclosure are shown in Figure 6.

CONCLUSIONS
I must congratulate the four students at Portland University, who developed the Python program to convert the complex SoundFont file format into something that the Teensy could use, and to write the actual Teensy Audio library routine itself. I’ve written several Teensy Audio library routines, and can appreciate how complicated they are to understand and write.

I think that the versatility of this library was limited by its ability to emulate only one musical instrument voice. The modifications that I made to allow for multiple voices, loaded from an SD card, make the whole concept much more useful. Also, my addition of Mod wheel and Channel Aftertouch capability is something that most musicians would require.

The 7-band Graphic EQ is also a nice feature to have. Having used such an EQ (one is included in my professional-grade digital mixer), I acknowledge that the implementation found in this program, using only a four-section biquad filter per band, is not as sonically accurate. But in my opinion, it works reasonably well.

Ten years ago, it would have been unthinkable to get an MCU as powerful as the iMXRT1062 on a module that costs only $20. The Teensy 4, with its associated Audio library, will certainly see a lot of use in many different audio projects. 

RESOURCES

References:
[1] “Build a SoundFont MIDI Synthesizer (Part 1): Using Teensy 4” ( Circuit Cellar 358, May 2020)
[2] “Fancy Filtering with the Teensy 3.6: Arm-ed for DSP” ( Circuit Cellar 346, May 2019)
[3] Teensy Forum: https://forum.pjrc.com/
[4] Code for this article is in the Part 1 (May 358) folder on the Circuit Cellar code and files webpage.

Teensy 4 MCU module and PT8211 Stereo DAC module:    PJRC  L.L.C      pjrc.com

AudioSynthWavetable object library and associated utilities:

Project repository located on GitHub at:
https://github.com/TeensyAudio/Wavetable-Synthesis

Teensy Audio System Design Tool:
https://www.pjrc.com/teensy/td_libs_Audio.html

NXP iMXRT1062 MCU Datasheet (manual):
https://www.pjrc.com/teensy/IMXRT1060RM_rev1.pdf

SoundFont Technical Specification:
http://freepats.zenvoid.org/sf2/sfspec24.pdf

WAVE PCM Soundfile format:
http://soundfile.sapp.org/doc/WaveFormat/

Microchip Technology | www.microchip.com
NXP Semiconductors | www.nxp.com
PJRC | www.pjrc.com
Princeton Technology | www.princeton.com.tw

PUBLISHED IN CIRCUIT CELLAR MAGAZINE • JULY 2020 #360 – Get a PDF of the issue

Sponsor this Article