With RF Comms and PIC32
Today’s microcontrollers have rich resources for implementing gesture control and other functions. Learn how these three Cornell students created a rhythm-based gesture game. The game has a base station where the main game control is run, and a module that runs on the player’s hand. Microchip PIC32’s MCUs are used to implement the game.
Our decision to work on this project was initially driven by the recent trend toward more active forms of entertainment, such as those provided by motion-control systems and virtual-reality headsets. We set out to create a similar type of device. We were further inspired by the games Rhythm Heaven and Dance Dance Revolution, in which the different physical inputs made by the player work to create a rhythm to which the player essentially dances. With all that in mind, we formulated Groovy Times, a game in which the players are prompted by on-screen directions to move one of their hands a certain way, thereby creating a rhythmic dance with their motions.
The system has two parts: a base station and a corresponding glove for the player’s hand. The main system controls are at the base station, where a user starts the game and selects a song. The user also wears a belt and glove, which has an accelerometer to determine the position of the hand the user is waving. The two modules communicate using two separate RF modules. A block diagram of the system is shown in Figure 1.
SYSTEM OVERVIEW
The precise game functionality makes the user tilt the gloved hand toward the direction of the arrow that appears on the screen. Upon reaching the correct location within the given time frame, the player gets a point and then proceeds to the next on-screen arrow. After a certain period of time, the game moves forward, even if the player does not reach the specific hand direction, and the player does not get points for the motion. This meant we needed to allow the player to be completely mobile in our hardware, so that actively moving would be a fun aspect of the game. These design goals led us to create the overall setup of the game: a base station where the main game mode aspects occur, a glove that contains the accelerometer for this purpose, and a connection from the glove to a belt, where the accelerometer data will be processed and sent using RF modules to the base station.
The game begins when the user is prompted to start the game with a click on a regular switch button functioning as a “select” button. The user can then select the song to play, using a potentiometer that can be rotated to create a scroll functionality. Once the user has scrolled over the desired song, the “select” button is used to choose it. Each song has varying levels of difficulty, which are controlled by how fast the user has to change positions and by the difference between the positions.
Once the game starts, the user has a certain amount of time to reach the desired position on the screen, at which point a green arrow or red check mark appears. To ease the user experience, the current position value transmitted from the glove is displayed at the bottom of the screen. This user experience is shown in the Figure 2. Different sounds signaling a correct or incorrect motion correspond to the sounds played. When the game ends, the user’s score is displayed on the screen, and the user has the option to start the game again.
The accelerometer, which is located on the glove, initially sends the data to a Microchip Technology PIC32 microcontroller (MCU) located on the user’s belt. The PIC32 then calculates which position the accelerometer data correspond to and sets the current position to a binary value associated with each position, such as left or right, based on a set threshold value. This data then is sent back to the base station. We selected an RF module to perform this action due to the reliability of its functionality. We used an Arduino as the device driver for the system, because of the availability of a highly functional library. The base station then receives this position-associated data through the same Arduino and RF module. This data was then smoothly integrated into the overall game state, as described above.
— ADVERTISMENT—
—Advertise Here—
To account for the unique values each user has for different positions, we implemented a calibration mode for the game that occurs at the start. When the game begins, the user presses a button to set the baseline values for all the different locations the user will reach throughout the game.
HARDWARE DESIGN
The hardware design had two key elements: the base station and the belt hardware. The key design decisions were based on ensuring reliability and ease of use. The elements of the belt hardware are the RF module, Arduino, accelerometer and PIC32, including their communication setup, as shown in row A of Figure 3.
The main sensor used is the Xtrinsic MMA8451Q 3-Axis Accelerometer from NXP Semiconductors, interfacing with an Adafruit breakout board. The accelerometer is placed on the back of a glove, worn on the player’s left hand, and connected with additional wires to the accelerometer to the PIC32 module. This interface records the orientation of the person’s hand in the X, Y and Z axes. For the I2C hardware, we added pull-up resistors to both the SDA and SCL lines, to ensure proper functionality of both lines. The schematic of this setup is shown in Figure 4.
The accelerometer is connected to the PIC32 MCU to send the data values over I2C. The PIC32 connected the Arduino to communicate the orientation of the hand. We selected a parallel method of communication of 4 bits and handshake method of communication for simplicity and ease of use. Additionally, the 3.3V output from the PIC32 was easily readable from the 5V Arduino.
The next stage of the belt hardware was the Arduino, which functions as the interface between the PIC32 and the RF Module (RFM69HCW) from Adafruit. The Arduino had a simple SPI connection to the RF module, which used MISO, MOSI, CS and some additional pins necessary for support.
Both the Arduino and PIC32 are powered by two separate 9V batteries attached to the belt, to make the entire system wireless. We selected two batteries to separate the part of the system with the Arduino, from the part with the accelerometer and PIC32. This separation additionally helps limit the number of wires traveling across the belt.
The base station hardware consists of four major hardware items, shown in Row B of Figure 3: the PIC32, a breadboard for all of the wiring, an Arduino and the same RF module described in the belt hardware section. The RF Module (RFM69HCW) is connected to the Arduino through SPI, using the same pins as previously described for the belt. The Arduino-to-RF module connection is shown in Figure 5. The Arduino outputs to the main PIC32. The PIC32 functions at 3.3V. The Arduino functions at 5V, so we use voltage dividers, to allow the PIC32 to receive a proper voltage level for the pins to read, and for protection of the pins, as shown in Figure 5.
The PIC32 is mounted on a PCB [1], which allows for connections to several separate components. A TFT, which is used for the display of the graphics for the project, is mounted on this PCB. Additionally, the PIC32 is connected to a button that moves through a game state, a potentiometer that is used to scroll through the various screens, and an audio jack output, which is used to connect to the speakers and to the DAC output. Each of these pin connections, except for the DAC connection, is connected through a 330Ω resistor to protect the pin from high current inputs.
SOFTWARE DESIGN
The code is separated into four different code bases: the base station PIC32, which runs the main game logic; the base station Arduino, which receives the accelerometer data; the belt Arduino, which transmits the accelerometer data; and the belt PIC32, which processes and converts the accelerometer data.
— ADVERTISMENT—
—Advertise Here—
The main code organization occurs using a state machine, which runs through three main states of main start screen, song select and game play. To set up the different states, protothreads [2] are used. Each game state is allocated its own thread, which is scheduled when the state variable is set to the state’s value. At the end of game play, the game resets to the main start screen, to allow for continued rounds of game play. Along with these threads, two other threads are continuously scheduled regardless of game state: the Input Controller and Sound Controller. Each of the threads uses the TFT library [3] to set up the necessary graphical display for the stage.
The main method of the code is responsible for setting up the ADC, SPI channel used for the DAC and any additional variables necessary for control of the game. This is where all of the threads are scheduled in a while loop. Direct Memory Access (DMA), which is used to play sounds, is also set up within the main method. A primary thread is responsible for all the graphics of the main start screen. This thread will yield every 700ms, to ensure that the thread responsible for the sounds at this stage of the game can be played.
The second thread controls the selection of the game play. During this game state, a user-controlled “knob” is set up, and the song-select graphics occur. A scrolling action on the screen is implemented by reading the analog value from the potentiometer. Every range of values corresponds to a specific song value on the screen, in ascending order. An external button is the triggering point to transition into the game-play state and to select the song.
The third thread is where all the game-play logic is implemented, and it is the thread where the most time is spent. This thread takes the moves from the two header files, and converts them into a graphic on the screen, as previously described. Additionally, this thread continuously checks for the player’s moves, and displays the results on the screen. An integer is used to continuously keep track of the score. Upon termination of the game-run state, the thread is de-scheduled, and the initial main menu thread gets rescheduled.
A separate sound controller thread controls all the sound output from the PIC32. Different sounds are played when a player misses, gets the correct point or a new sound is played. Each of these flags triggers the output of a different sine table to the speakers for approximately 1 second, allowing sound effects of different pitch to be emitted. Three separate sine tables of widths 256, 128 and 64 are used for the different sounds. All sound is sent to the DAC via DMA. We then enable the DMA channel with DmaChnEnable() and yield the thread for 5ms at a time. A timer is repeatedly checking Timer 3 to see if around 1 second has passed (50,000 ticks). Once it has, the DMA channel is again disabled to stop the sound and the thread reset, then yielded until another flag is set.
Several auxiliary header files support the different modes of game play. For ease of use, we define the two different typedef structures—song and moves. The song typedef contains the name of the song and the series of moves described by the moves typedef. The moves typedef contains the direction of the move, the time when the move should occur, and how long the move should be displayed. Following this, a series of moves is stored in other header files to constitute a song. An additional header file contains all the songs that will be referenced.
BELT MCU CODE
A separate set of code runs on the belt MCU. There are three separate threads, represented by three separate states (Figure 6). The calibration state is the initial state, where a value is calibrated to each of the positions at which a user may be positioned. The “I2C” state is the general state where the value is being read from the accelerometer. The transmission state is where the accelerometer data is being converted into a binary value that corresponds to a position of the hand (up, down, left, right or none). The calibration state only gets scheduled during the first iteration of the code, when a flag is set to indicate that calibration has not occurred. Following this, there is round-robin scheduling of the I2C state and transmission state.
The calibration thread saves a set of values to every direction at the press of a button. Once a press is determined, the calibration automatically moves to the next direction. A counter is used to iterate through the directions. At every press, “1” is added, which indicates the transition at the next state. Once the calibration state is finished, a flag is set, indicating that it should no longer be scheduled. Debouncing is done on the button to ensure that there are no false transitions between the states.
The I2C code is modeled after the code by Ritchken and Liu [4]. The thread is simple, because it is responsible only for continuously reading from three separate registers in the accelerometer, which correspond to the X, Y and Z directions. These values then get set to global variables that will used in the other state.
The transmission state determines to which direction the accelerometer data corresponds. To do this, the code checks to see if the data matches the values for the direction within a certain range, to account for variability within the data and the positioning. This is done using a series of conditional statements. Once the direction is determined, a binary value corresponding to that direction is sent to the Arduinos by outputting a low or high value for each corresponding bit to the Arduino.
The final aspect of the code is the RF transmission and receipt of the code, which is done on Arduinos, respectively. This code was done on an Arduino to be able to use the Radiohead library that is available, most specifically the RH_RF69.h file [5]. The baseline for this code was the RX and TX demo code found within the Radiohead library. The key setup was similar for the code on both the transmitting and the receiving side. Within this section, we set the frequency to 915MHz and initialized the entire serial setup. We set up serial primarily for debugging purposes, until we could confirm that data would be reliably transmitted and received. This aspect of the code would additionally check to ensure that the RF module was properly initialized, and to run a test on the RF module to ensure functionality.
After the initial steps, the two Arduino codes diverged in their setup. On the transmission side, we input the data given by the PIC32 data and converted this into a transmittable game state. Then a logic statement was used to convert the state into an integer value representative of the game state. We added an additional error state for transmission, for the cases when values should be discarded. The transmission of the data also occurred in the main loop and went through the following stages: sending the data, waiting and checking if the data was received.
Based on the demo code [5], we would only check for a confirmation of the receipt of the data. The data was transmitted in packets of a 6-bit char array titled radiopacket in the code. Because of the fast-paced nature of the game, we decided that it would be more effective to move onto the transmission of the next potential game state if a transmission was not received, rather than focusing on retrying to send a packet that was not received.
The receiver code starts with the same setup, but with the additional setup of pins as outputs to the base station PIC32. In the main loop, every time there is an available packet of the correct length, the packet is received and saved. Following this, all the transmission states are converted, using a large case statement to the output to the main board PIC32, and then the values are outputted.
RESULTS AND FUTURE WORK
In the final version of our project, we had a fully functional game with a single glove working, which can be seen on a demonstration video [6]. Five different songs corresponded to different modes of game play. Each of the modes translated to a different level of difficulty and increased in complexity throughout the duration of the songs. The transmission of the data from the glove-mounted accelerometer to the base station occurred fast enough that the game state was responsive within the time frame of a certain arrow being displayed on the screen. The on-screen feedback displaying the position of the glove was fairly accurate, which corresponded to accurate transmission of glove positions. Extensive testing was done on each of the components to get to this end result.
— ADVERTISMENT—
—Advertise Here—
To confirm our results, all our group members tested several rounds of game play. Occasionally, some players had difficulty reaching the precise location of a certain position. However, we attributed this difficulty to game play. Sometimes players also had to move other parts of their bodies to hit a certain position, which added to the mobile aspect of the game. Additionally, the game tended to train the players, and players typically improved in the game after several rounds. To ensure that the usability of this game met our and others’ standards, we asked several classmates to play. The game overall had a high usability and provided mild aerobic exercise from the movements.
Prior to game play and the assembled product, we tested each element of the game individually. The results of the individual stage mimicked the final stages of the combined elements. The final state of the accelerometers allowed us to determine the direction of a player’s hand movement within a significant enough spatial range that a player could hit a direction state without the ranges of those states overlapping or becoming too large. After extensive testing, the optimal calibrated ranges of each of the given directional states was defined.
The final state of the RF transfer performed with high accuracy. A packet was sent out without confirming delivery only about 1 in 40 times. To confirm that the correct data was received on the base station design during game play, we displayed the data on the screen. The entire system of accelerometer reading to data transfer performed very smoothly in the end. This can be seen in the video of our demonstration [6]. We believe the overall outcome of our project to be a success because it was fully functional. We can provide several updates for the game. Future improvements of the game would include the addition of a second glove and accelerometer, to allow for more complexity within game play. 
RESOURCES
References:
[1] Land, Bruce, and Sean Carroll. “Development Boards.” Cornell University ECE 4760, 2019, people.ece.cornell.edu/land/courses/ece4760/PIC32/target_board.html.[2] Dunkels, Adam, and Oliver Schmidt. “Protothreads.” Protothreads – Lightweight, Stackless Threads in C, dunkels.com/adam/pt/.
[3] Land, Bruce. “Adafruit TFT LCD Display Model 1480 and Keypad .” Cornell University ECE 4760, 25 July 2017, people.ece.cornell.edu/land/courses/ece4760/PIC32/index_TFT_display.html. [4] Ritchken, Brian, and Jim Liu. “Blimp-F-O.” ECE 4760, 2015, http://people.ece.cornell.edu/land/courses/ece4760/FinalProjects/f2015/bjr96_jl2628/bjr96_jl2628/bjr96_jl2628/
[5] Ada, Lady. “Adafruit RFM69HCW and RFM9X LoRa Packet Radio Breakouts.” Adafruit, Adafruit, 2016. learn.adafruit.com/adafruit-rfm69hcw-and-rfm96-rfm95-rfm98-lora-packet-padio-breakouts. [6] Video of fully functional game:
Parts Used:
1) Adafruit TFT Display: https://www.adafruit.com/product/358
2) Microchip PIC32 Microcontroller: http://ww1.microchip.com/downloads/en/DeviceDoc/61146B.pdf
3) Adafruit RF Module: Adafruit RFM69HCW
4) Arduino Uno: https://store.arduino.cc/usa/arduino-uno-rev3
5) Adafruit Accelerometer Breakout: https://learn.adafruit.com/adafruit-mma8451-accelerometer-breakout/downloads
Adafruit | www.adafruit.com
Arduino | www.arduino.cc
Microchip Technology | www.microchip.com
NXP Semiconductors | www.nxp.com
PUBLISHED IN CIRCUIT CELLAR MAGAZINE • APRIL 2020 #357 – Get a PDF of the issue
Sponsor this Article
Kristina Nemeth is a Master of Engineering student studying Electrical and Computer Engineering at Cornell, where she also completed her undergraduate degree. Her interests include digital electronics and power electronics. In her
free time, she works on Cornell’s Autonomous Underwater Vehicle project team.
Joshua Diaz is a firmware engineer at electric vehicle startup Seres. He graduated from Cornell University in 2019 having studied Electrical and Computer Engineering. Joshua likes to spend his free time making his own indie
games for Mango Snoopers Game Studio.
Jennifer Fuhrer is a device engineer for the United States Department of Defense. She graduated from Cornell University with a B.S. in Electrical and Computer Engineering. In her free time, she paints acrylic and watercolor
landscapes.
