Build a Play-Along Keyboard

Using a PIC32 Microcontroller

Playing piano often takes years to learn. These three Cornell undergraduates embedded LEDs into the keys of a play-along keyboard and used a PIC32 as the interface to guide users note by note through songs. Sheet music for the songs was shown on a TFT display.

The core of our project consists of an Alesis Melody 32 commercial 32-key keyboard and a PIC32 development board Figure 2 [1] containing several peripheral devices. The heart of the development board is the PIC32MX250F128B microcontroller. The MCU is connected to an MCP23S17 port expander [2] for extra input/output pins, and an Adafruit Model 1480 TFT LCD display. The development board also contains headers connected to every pin of the PIC32, a power connector, and a reset switch. The TFT display and port expander communicate with the microcontroller over a serial peripheral interface (SPI).

The microcontroller primarily interacts with the LEDs and the keyboard keys. At a high level, the microcontroller turns the embedded LEDs on and off at appropriate times by communicating with four shift registers, and detects key presses by reading row and column lines from an existing key matrix.

Using a standard format such as MIDI initially seemed appealing for storing and representing song files; however, our desire to display the songs as sheet music made most standard formats difficult to use. We decided for this reason to use our own format and wrote a Python script to translate between a format that is easy to write and a data structure that is easy to work with in code.

INTERFACE

On reset, the TFT display presents the main menu with a list of song choices and the Freeplay mode option. The user navigates within the menu and makes a selection, using the three leftmost keys on the keyboard. If the user selects Freeplay mode, no music is displayed; instead, when a key is pressed, the associated LED illuminates. We added this mode for debugging purposes but retained it as a feature. If the user selects a song in the main menu, the TFT display shows the first two measures of the song (with the current note and chord highlighted in red), and the LEDs for the keys required to play those notes illuminate. When the user presses all the lit keys at the same time, the next set of keys lights up, and the next note on the sheet music is highlighted. This process of pressing the highlighted keys continues until the song is finished, at which point, after a brief delay, the keyboard returns to the main menu. The user can also press the reset switch at any point to exit the current song and return to the main menu.

We modeled and 3D printed an enclosure and mount for the TFT display. We originally planned to mount the display in the center of the keyboard but settled on the left side closer to the PIC32 board, due to the wire length. We cut a hole in the keyboard’s case using a Dremel tool to allow the display cable to pass through and then attached the display enclosure with hot glue and electrical tape.

For the reset switch, we soldered two wires to the existing reset switch on the development board and passed them through a hole in the keyboard case. Then we soldered those wires to an external switch, which we taped to the back of the case. Since the keyboard case matched the electrical tape we were using, we were able to quickly attach the switch and display enclosure without sacrificing too much in terms of aesthetics.

LEDS

Our project contains 32 LEDs, one embedded into every key. To mount the LEDs, we cut away some plastic ribbing in the center of each key, drilled a hole for the LED, and secured the LED with hot glue. The LEDs need to be wired to the shift registers according to the schematics shown in Figure 3. We first soldered wires to each lead of the LEDs, covered the connections with heat shrink, and twisted groups of eight positive or negative wires into bundles to make the wiring more manageable.

We separated the circuitry onto two perf boards. One board contained an array of 470Ω resistors (Figure 4). For this board one lead of each resistor is connected to the ground and the other resistor lead is connected to the negative lead of an LED. On the other perf board, we wired our four SN74HC595 shift registers following the schematics in the datasheets [3] and referenced below. Each positive lead of the LEDs connect to the appropriate shift register output. While we could have mapped each key on the keyboard to any shift register output in software, it made sense to keep them in order, in hardware.

The first shift register is connected to the LEDs associated with keys 1 through 8. The first shift register’s QA output is for key 1, and the QH output is for key 8. The next three shift registers are wired for keys 9 through 16, 17 through 24, and 25 through 32. We accidentally swapped the wires for two LEDs and had to patch this in software by setting the value for LED 13 to that of LED 14 but normally this would not be required.

Each shift register has three inputs that connect to the PIC32: the shift register clock (SRCLK), storage register clock (RCLK), and data input (SER). All four shift registers share the same RCLK and SRCLK clock signals. The data inputs are unique: the four SER shift register inputs are connected to DATA0-DATA3, respectively, as shown in Figure 5. There are a total of six connections needed between the LED subsystem and the PIC32 A six-wire ribbon cable connects the shift register perf board (Figure 6) to the port expander pins on the PIC32 development board.

Why do we need a port expander? The PIC32 has only 19 GPIO pins, five of which are used by the TFT display. Since we need six GPIO pins to drive the LED circuit and another 13 pins for detecting key presses, (discussed later), we need the port expander for the additional GPIO pins. The port expander, itself, uses an SPI interface that requires four GPIO pins, and two for interrupts, but effectively gives us another 16 in return—enough inputs/outputs for this project.

In software, we use a 32-bit number as a bit mask to represent the intended state of the LEDs. We wrote a function to update the shift registers based on that bit mask. The function loops eight times, and within that loop, sends 1 bit of data to each of the four shift registers. It first clears the active data pins and adds each of the four data pins to the list of active pins if the associated LED is supposed to be on. We then wrote the active pins to the port expander using the SPI channel, flipped the shift register’s SRCLK and RCLK lines twice to toggle them, and then cleared the port expander.

We added a 100µs delay between writes so that we don’t write faster than the shift register can process the inputs. This sequence of writes to the port expander sends a 1 or 0 to each shift register if that output is supposed to be on or off, and then pulses SRCLK so the shift register stores that bit of data. After eight iterations of this loop, the function toggles the RCLK pin, after which time the shift registers will have the desired outputs.

KEYBOARD INPUT (SOFTWARE AND HARDWARE)

The next major step was to detect key presses from the keyboard. We had originally planned to set up a key matrix driven by the PIC32, with four columns and eight rows. Each key would be connected to a unique row and column pair and would connect the row/column when it is pressed. The PIC32 would pull high one column at a time, and by reading the output of each row would be able to detect if any keys connected to that column were pressed, by checking whether the associated row was also high. The PIC32 would do this rapidly for each column, and therefore be able to detect key presses.

However, when we received the keyboard and took it apart, we found that it already had a similar key matrix implemented. The existing matrix used five columns and eight rows, and was an active low system, that is, the keyboard MCU pulled columns low when they were, by default, high. Rather than adding our own, we spliced into the existing matrix by tracing the connections back to the keyboard motherboard and soldering ribbon cables to those pins.

Since we were no longer driving the matrix, we had to change our approach to reading from it. We initially planned on using the change notification feature of some pins on the PIC32. This feature generates an interrupt when the logic level on a pin changes. We planned to connect the columns to these change notification pins, and within the interrupt service routine (ISR), read from the port expander to get the value of the rows at that time to determine which key was pressed. In practice, this did not work, because by the time the PIC32 had entered the ISR and read the rows, the keyboard had already pulled a different column low, and the data was worthless. Instead, we read the inputs from the GPIO pins to which the columns were connected and read the specific port within the port expander to which the rows were connected. This worked fairly reliably. There were still some issues of phantom presses or keys turning off, so we added a debouncing buffer that considered a key pressed only if four or more of the last 10 readings of that key were low. This fixed most of the issues.

SOFTWARE

The code for our project is available for download [4]. Our software contained a few important data structures and modules to interface with the hardware and the display. We controlled the LEDs through our software’s led.h module. This module stored the state of the LEDs as a 32-bit integer. It provided two functions to our main method—one to initialize the LEDs and one to set them. This allowed us to simply pass a 32-bit integer representing the status of the 32 LEDs into the leds_set function and let the led module handle the rest.

The keys.h module exposes the keyboard input to our main method. This module manages all the row/column mapping and hardware polling and exposes only three simple functions: an initialization function, a polling function, and a wait for release function. In the initialization function, we set up the PIC32’s PORTB GPIO pins as our columns for the key matrix. We then set up the rows as I/O pins on the port expander’s Port Y. We send the port expander setup data over an SPI channel. The polling function polls the keys and returns a 32-bit integer representation of whether each of the 32 keys is on or off. Finally, the wait for release function tells the software to stop continuing the song until the currently held keys are released. This prevents a single keypress from the user from being interpreted as multiple correct repeated notes by the software.

This function is only necessary when navigating the main menu (Figure 7) since we wanted our menu to scroll just once per key press.

A more complex player module manages to play a song and display the sheet music. This player module exposes two functions: song_start, which starts playing a song, and player_update, to update the currently playing song. The player_update function must be called often if the player is expected to be responsive; it manages writing to the display, polling the keys, and updating the LEDs a single time. The fact that our player module depends on an outside source consistently calling player_update to function properly is not a great modular design choice, but it worked for our needs. Were we to continue work on or extend this project, we could refactor this module to remove this dependency.

At the highest level, our main function implements a state machine containing three simple states: MENU, to represent the main menu selection; FREEPLAY to indicate our available Freeplay mode; and PLAYING, to represent when a song is being played. Our main thread then loops through this state machine, checking each state and performing the corresponding actions. The main menu is simple and intuitive. The list of available songs is shown in Figure 7, and using the bottom three labeled keys on the keyboard you can tab up, tab down, or select. When a song is selected, the player module is called, and we switch into the PLAYING state. Our PLAYING state simply calls the player_update method and checks the return. If the player is finished, it returns a 1. Then the main thread waits for 3 seconds before exiting the main menu because a brief interval before exiting the song looks nicer for the user. This also causes our player_update function to be called for every iteration of the loop, which is about as often as possible.

Originally, we chose to leave the TFT blank during our Freeplay mode, but to ensure that the TFT was working during this mode, we decided to display a placeholder background trapezoid. We are very proud of how easily our modular design allowed elegant implementation of our Freeplay mode. We simply call the leds_set function with the keys_poll function as the argument. In one line of code, the LEDs are set to reflect exactly what keys are pressed, in real-time. We had originally implemented this to test that both the keys and the LEDs were mapped correctly, but we also thought that it could be a fun feature for the user.

SONG REPRESENTATION AND SHEET MUSIC

We gave careful consideration to how we represented the music in software. Some musical background is needed to understand why we made these choices. We will go over some of the basics here but have also added articles about musical notation in the resources section. We wanted to do two things for each song: display the notes on the piano, and display the sheet music (Figure 8) on the TFT.

In Western music, and on our piano, there are 12 notes labeled A through G, along with 5 intermediate notes called flats and sharps (for example, C sharp or C#). These 12 notes make up one octave. The 12 notes are also labeled by a numeric octave suffix. A G3 note, for example, is exactly twice the frequency of a G2 note. Our piano contained the notes F3 through C6 or 32 keys. Most simple songs can be represented in two parts. The “melody” is the part of a song you might sing: “Row, row, row your boat, gently down the stream.” The “harmony” is the set of notes in the background that you could imagine being played as accompaniment. On the piano, we can play both parts. In most simple songs, one group of notes (a chord) is played by the left hand for the harmony, and one note is played by the right hand as the melody.

Given this background, we decided we needed a format in which, at a specific time, we could simultaneously represent the notes in both the chord and melody. Melody notes were represented with their letters and octave numbers, and the chord was represented as a group of notes. We did not have space in the sheet music to display the individual notes in the chords, so we decided to display the sheet music for the melody only, and to display the chord name above the music (Figure 9). The individual keys for the chord are still displayed on the keyboard. Representing chords in this way is common in sheet music.

In sheet music, the timing of the notes must also be represented. Notes are generally split into divisions of two. A quarter note is as long as two eighth notes, a half note is as long as two-quarter notes, and so on. We decided that we only needed to represent notes as small as 16th notes and, as such, decided to represent the timing of our notes in what we call “quanta”—one 16th note in terms of timing.

We put everything we needed for the piano key LEDs and sheet music representation into a quanta C struct, which contained the starting time, melody note name, melody note duration, chord name, and notes in the chord. We also created a C struct representing a song with an array of quanta.

We “wrote” our songs to a text file delimited by line breaks. The first three lines were the song title, the beats per minute (currently unused), and the octave number. The octave number indicated to shift the sheet music representation by one or more octaves. This allowed a wider range of notes to fit in the limited height of the TFT display.

After the initial lines, each line starts with the quanta that the line represents. Because these quanta are sixteenth-note divisions, we wrote the quanta number in hexadecimal followed by a comma. Next is the chord name, the duration of the chord (in quanta), and all the notes making up the chord, followed by a comma. The line ends with the note in the melody and its duration. The chord information may be left out on some lines because some notes don’t need a chord.

With this representation defined, we wrote a Python script that translated each line into an initialization of C quanta, and then finally initialized the whole thing into a C song. Currently, the structs are stored as constants in program memory. This was not an issue, given the small number of songs that we wrote. However, with larger and/or more songs, this method of storage would have to be reconsidered.

All the information that we needed to display the proper keys and the sheet music was thus provided. With a simple sheet music graphic library that we wrote, we could display flats, sharps, and all types of notes by using the information we had from each quanta. However, for musicians reading this, the system had limitations. We defaulted to writing all the music in the treble clef, in 4/4 time, and with no key signature. We also did not have support for rests, since many basic beginner songs do not need them. Although most of these things could be readily handled with the format we have set up, we felt it was still important to acknowledge the limitations.

Within the player, we are now able to step quanta by quanta toward the end of the song. During each quanta, we highlight the correct notes in the sheet music and light up the correct LEDs. We then poll the keys until all necessary keys are pressed, and then move on to the next quanta. This process repeats until the song is over— and now, you are a musician!

CONCLUSION

In many ways, our project came together better than we had expected. We started the project excited about interfacing with the keyboard hardware and creating a fun musical tool but were unsure how far we would get in our ability to represent songs and generate sheet music. With careful initial planning, we were able to make an elegant system that feels smooth to play and does a good job of representing simple sheet music.

As we continued work on this project, we found that adding features not initially planned for inclusion, such as Freeplay mode, sheet music for note duration, and sharps and flats in the sheet music were so easy to add that it felt wrong to not add them. In the end, we were happy with the work we did, and with the additional features, we were able to include because of our initial design choices. 

Part	Part Name	Vendor Used	Purchase Price
LEDs	EDGELEC 5mm LED	Amazon	$6.19
Resistors	470 Ohm Resistor	Amazon	(included with LEDs)
Shift Register	SN74HC595	Amazon	$4.84
Keyboard	Alesis Melody 32	Amazon	$50.15
Development Board	PIC32 Big Board	ECE 4760 Lab	$10
TFT Display	Adafruit Color LCD Model 1480	ECE 4760 Lab	$10

REFERENCES
[1] PIC32 Users Guide: http://ww1.microchip.com/downloads/en/devicedoc/61146b.pdf[2] Port Expander MCP23S17 Data Sheet:
https://ww1.microchip.com/downloads/en/devicedoc/20001952c.pdf[3] Shift Register Datasheet: SN74HC595
https://www.ti.com/lit/gpn/sn74hc595[4] GitHub repository containing our source code
https://github.com/MatiasGoldfeld/play-along-keyboard

RESOURCES
How to Read Sheet Music
https://www.musicnotes.com/now/tips/how-to-read-sheet-music/

Music Theory—Basic Notation
http://openmusictheory.com/basicNotation.html

PUBLISHED IN CIRCUIT CELLAR MAGAZINE • SEPTEMBER 2022 #386 – Get a PDF of the issue

Sponsor this Article