EEG Error Correction Interface

Anxiety Takes Control

The engineering of Brain-Computer Interface (BCI) devices is a fascinating topic. EEG measurement is a key tool in these endeavors. In this project article, these three Cornell students built an EEG error correction interface, in which users influence which way a conveyor belt travels, based on their calculated anxiety levels.

We chose the Muse headband over other commercially available EEG-measuring devices, because of its price, considering its number of available electrodes. It has four main electrodes located on the forehead and behind the ears, and three additional reference sensors. The wireless nature of the headset also gave us confidence in the safety of the system. Additionally, the Muse has an easy-to-use interface that allows users to obtain both raw data and pre-calculated values—useful for analysis—from the headband. This project makes use of the built-in calculated values and raw data from the Muse to demonstrate that users can use EEG signals from a limited number of sensors to correct errors in machine behavior.

Our project was modeled as a trial-based “game.” Below one end of our conveyor belt was a solid-colored green box, and at the other end was a dotted yellow box. In each trial, a solid-colored green block or dotted yellow block was placed in the middle of a conveyor belt. The user’s goal was to get the block to drop into its matching patterned/colored box. In each trial of the game, the conveyor belt began to move in a randomly chosen direction. If the random direction was correct, the user needed to stay calm to allow the block to travel to and fall into its collection bucket. If the random direction was incorrect, the user needed to correct the direction by spiking his or her anxiety level, as determined by EEG data collected from the Muse Headband. We hypothesized that as the block traveled toward the incorrect box, user anxiety would instinctively spike higher, thus creating a natural way for the user to correct machine behavior.

We determined user anxiety levels according to theories from current neuroscientific research. Studies have shown that certain frequency bands of EEG data can correlate with mood and concentration levels. Anxiety is thought to correspond to activity in the Beta and Gamma frequency bands, which range from 13-30Hz and 30-44Hz, respectively. We found the cleanest data at Gamma frequencies, and therefore looked for spikes in activity in that band to determine the anxiety level of users [1] [2].

OVERALL DESIGN
We designed a system to get a user’s EEG data in real time to a PIC32 MCU. We processed these data on the PIC32, then used the result to determine whether a user was anxious enough to change the direction of the conveyor belt. In our system, data were sent over Bluetooth from the Muse headband to a phone [3]. Then, both raw data and pre-calculated values outputted by the Muse interface were transmitted via OSC (Open Sound Control) to a computer, where they were pre-processed and sent over serial to the PIC32.

OSC is an open-source, application layer protocol that is message-based and optimized for communication between multimedia devices. OSC operates using UDP (User Datagram Protocol), and sends packets from the OSC Client to the OSC Server [4].

A Fast Fourier Transform (FFT) was performed on the raw EEG data, once received from the computer (Figure 1), to determine general brain activity, while the pre-calculated values provided instantaneous information about potential spikes in the anxiety-related (Gamma) frequency bands. This allowed the PIC32 to normalize data from the headband, and quickly compare the normalized data to instantaneous anxiety values. Simultaneously, the game-play logic responded to button pushes to start the game, by moving the conveyor belt in a random direction. The conveyor belt switched directions if the user’s anxiety levels (Gamma magnitude) surpassed general brain activity by a certain threshold. Figure 2 is a block diagram of the system’s high-level components.

HARDWARE DESIGN
Our hardware design had three parts: serial hardware, motor control hardware and game-play hardware. For serial, we used a serial-USB cable (Adafruit Product ID 954). The UART receive pin was wired to RA1 on the PIC32, and the UART transmission pin was wired to RB10 on the PIC32 (Figure 3). Additionally, the serial-USB cable was grounded to the MCU ground, and the USB V_CC was not wired to anything.

The motor we used to control the movement of the conveyor belt was a continuous rotation parallax servo. Capacitors from 5V to ground and from CPU signal to ground stabilized the power line and motor control signal. The game-play hardware consisted of three buttons and three LEDs. The buttons corresponded to Start Game (grey button), Correct Bin (blue button) and Incorrect Bin (red button). The Start Game button was wired to RA2 on the PIC32 and ground, the Correct Bin button was wired to RA3 on the PIC32 and ground, and the Incorrect Bin button was wired to RA4 on the PIC32 and ground. The LEDs corresponded to whether or not the player got the block in the correct bin, as well as which bin was correct.

We had several design considerations pertaining to the mechanical construction of our conveyor belt. Because eye movement can be observed in EEG measurements, we ensured that the horizontal length of the conveyor belt was relatively short (around 12″), so that when the user’s eyes followed the traveling blocks, the EEG data would not be affected. Additionally, we picked a continuous rotation servo with low torque (38 ounce-inches), because we did not need the belt to move at a high speed.

Knowing that our project required lots of iterative testing, we used 2020 aluminum T-slot tracks for the conveyor belt frame, for durability and universal hardware mounting. Last, we used multiple rubber bands as the belt, to keep our costs low and allow easy replacement during testing. A digital model of the conveyor belt plan and the final constructed conveyor belt are shown in Figure 4.

SOFTWARE
Our software for the PIC32 was designed to optimize both the consistency with which we could detect anxiety spikes and the transferability of the system among a wide variety of users. So, we therefore took a unique approach to determine anxiety. It integrated calculations performed within the Muse interface, and calculations performed by our software on the PIC32, using raw EEG data. The Muse interface calculations in particular were helpful for consistently and quickly reporting the magnitude of the Gamma frequency band. However, this calculation provided only a snapshot of the brain’s EEG activity. We therefore also took the raw data from the Muse to create a running average of general brain activity.

This running average theoretically was different for each user, and gave a good indication of baseline brain activity for individuals. By comparing the magnitude of the Gamma frequency band to the general brain activity, then using the difference between the two to determine anxiety states, we could rapidly and robustly detect anxiety spikes, while retaining the ability to switch between users at any time.

The PIC32 software was broken down into four threads: the timer thread, the motor thread, the serial thread and the FFT thread—along with an ISR. A Python script sent data from the headband to the PIC32 over serial [5]. On a high level, Gamma magnitude and all raw data recorded from the headband were sent over OSC to a computer. The computer was running a Python script that sent data to the PIC32 over serial. The PIC32 received these data in the serial thread, where raw EEG data were placed into arrays and Gamma magnitude data were saved into single variables. A PIC32-based FFT was then performed on the raw EEG data in the FFT thread.

The magnitudes of all frequency bands within a biologically relevant range were then averaged together and used to inform a running average of general brain activity. In the motor thread, this running average was then compared to the pre-calculated Gamma magnitude data and, if the Gamma magnitude data were much greater than the running average, an anxiety spike was marked as occurring. This changed the direction of the conveyor belt during a trial of a game. The details of each software component are explained below.

Python Script: The Python script first set up a server to receive the OSC stream of data from a phone. Once this server was established, methods were added to the server to receive data from specific transmission channels (that is, to receive specifically from the raw sensor data channel or the Gamma magnitude data channel). These methods added the received data to particular queues—either a Gamma queue or a raw sensor queue—given that the data actually contained a number as expected (instead of garbage).

Once the methods were added to the server, serial communication was established with the PIC32. We then used a multithreaded Python script to receive OSC data and add them to queues. Simultaneously, the data were preprocessed, some significant figures were truncated, and these data were sent with an appropriate tag over serial to the PIC32. We added character tags to all serial transmissions, in case we could not guarantee the order of data transmission. For our analysis, it was important to identify whether we were sending raw data or Gamma magnitude data, and the sensor from which the raw data originated.

Serial Thread: The serial thread saved both the raw data and the Gamma magnitude data sent from the headband via the Python script. Each time the Python script received an exclamation point from the serial thread, it sent data. Each time the serial thread sent an exclamation point, it prepared to receive data. In the serial thread, we used a switch statement, based on the transmission tag added in the Python script, to determine into which array or variable to save the data.

If the values received over serial were not in the correct range—meaning we received corrupt data—we checked the data’s character tag. Then we set the value in the appropriate array either equal to the previous value in the array, or the value of the Gamma magnitude variable equal to its previous value. Although this caused some loss of accuracy, we took this approach because EEG readings do not tend to change quickly, and the timing of our system was slightly more important than making sure every data point was exactly correct.

Ultimately, we decided to save only the raw data from the two Muse sensors located on the forehead for our running average FFT, because these were the two sensors on the Muse that contributed to the Gamma magnitude calculation, and we wanted to keep the sensors used consistent for both calculations. We created two arrays to collect the raw data for each Muse sensor. While one array was being filled, we performed an FFT on the other array. This ensured we never lost data, because the FFT could take fairly long to complete. We used a lock to ensure that we were filling the correct array.

MORE THREADS
FFT Thread: The FFT thread first checked if any raw data array contained 256 data points. If a set of arrays was full, it would perform an FFT on the filled arrays, while allowing the other, unfilled arrays to be used in the serial thread to collect data. We used an FFT algorithm that was optimized for use with fixed-point numbers. Once the FFT was complete, all FFT bins were averaged together, and this average was used to update the running average baseline of all EEG data for anxiety thresholding.

Motor Thread: The motor thread controlled the movement of the motor and the game-play logic. There were three buttons: Start Game, Correct Bin and Incorrect Bin. Correct Bin and Incorrect Bin variables incremented either the correct score or incorrect score, respectively, ended the current game and reset all the game logic. The Start Game button caused the conveyor belt to start in a random direction from a randomly generated seed, based on the current time.

We compared our saved Gamma magnitude value to our running average from the FFT’ed raw data, as completed in the FFT thread. If the Gamma value was above a set threshold for a given amount of time, then we considered this an “anxiety event.” When the Gamma value fell below the threshold for a given amount of time, it was considered “not an anxiety event.” Waiting for a certain amount of time before declaring an anxiety event ensured that the user had actually calmed down, instead of the PIC32 just receiving a transmission error. If there was not a valid game, meaning that the “correct” or “incorrect” End Game buttons had been pressed, the conveyor belt turned off.

We kept track of time to determine when we believed we had enough baseline EEG data to begin playing the game. We typically had enough data about a minute after transmission began. The ISR changed the motor direction by adjusting the length of the PWM pulse. The desired motor direction was set in the Motor Thread. In main, we initialized all the threads and scheduled them based on round-robin scheduling. We also initialized the motor and set the input pins. Additionally, we created the sine look-up table and the window array to help optimize algorithm execution time and scale the data values to the correct range in the FFT thread.

TESTING AND RESULTS
Serial Communication: We first tested our ability to communicate over serial. Our original implementation of serial caused us to get a large amount of unexpected and inaccurate data on the PIC32. To remedy this, we decided to send data only when the PIC32 was ready to receive, as indicated by it sending an exclamation point over serial to the computer. We then noticed that our serial communication was hanging at random times. To fix this, we cleared all serial errors each time a read or write was performed. This solved many of the hanging problems, yet we still could not reliably establish communication for more than about 5 minutes. We then began to flush the serial buffer in the Python script, and to lock our Python thread whenever we performed a read/write. This finally fully fixed the problem and allowed us to communicate for hours at a time.

We then noticed that, despite having a baud rate of 115200, we were becoming massively delayed in our serial transmission and creating queues of steadily increasing size, so that any data sent were delayed from real time by thousands of data points. Because our project relied so heavily on quick responses to fairly noisy data, we decided to average together three data points in the server methods, before adding data to a queue. This allowed us to keep up with transmission and keep our queue size between 1 and 0 at all times.

Speed and Performance: Once each thread functioned as desired and all data were properly transmitted, we began to analyze speed and performance. Given the size of our FFT, we ended up updating our running average about every 3 seconds. This was an appropriate rate, given that the average should not rapidly change from millisecond to millisecond. Instead, it should hold as a fairly steady baseline, once the program had been running long enough to collect several data points.

We then analyzed our speed in receiving the Gamma magnitude data. We updated these data slightly faster, about once every half second, allowing for many data points to be received during any given trial of the game. Because of this data rate, we adjusted our conveyor belt to move fairly slowly. This allowed for appropriate debouncing of anxiety spikes and ample time for users to respond to the conveyor belt motion to correct errors.

Thresholding: We then focused on thresholding. To achieve clean results, we added a debouncing component to detecting anxiety spikes. For an activity to be called an anxiety spike, an anxiety state needed to be held for four data points. Similarly, for a spike to be marked as complete, activity would need to be below the threshold for more than two data points. The threshold was determined by trial and error. However, by using the running average of all FFT’ed data as the baseline for this threshold, the game was relatively transferable to different players. A player with a higher baseline would need to cause a higher spike for an anxiety event to occur. After tuning the threshold, we found that our accuracy in getting blocks into correct bins was over 90% for data taken from one user over 60 trials. We are interested in further investigating this accuracy for different players, and taking more structured data for the original player.

Our final result was a functional EEG error correction interface that took input from a user’s EEG waves and adjusted the behavior of a conveyor belt accordingly. This project required a great deal of interaction with an external system and was highly dependent on visual cues. A video demonstration of the final product can be viewed here:

CONCLUSION
We were able to design, build and demonstrate machine error corrections in real time using EEG signals. In 60 trials performed by one user, the correct trial outcome was achieved over 90% of the time. Our design generally met our expectations. We were highly satisfied with our data analysis abilities and the cleanness with which we could ultimately label anxiety spikes.

In terms of future improvements, we would like to add a learning component to our game. If a user marks a trial as wrong, we would like to have functionality that can adjust our thresholding to reduce the likelihood of an incorrect round in the future. We would also like to further investigate how EEG data can be used in human-machine interactions, maybe by feeding EEG data into neural networks to see whether we could discriminate thoughts such as “left” or “right”. 

Author’s Note: This project used the Muse 2 EEG headband and the Mind Monitor app [3] to receive the raw sensor data and pre-FFT’ed Gamma data over OSC. The Python written for this report was based on the open-source project, “Telekinesis Car,” created by Devin Delfino [6]. This project was inspired by an experiment conducted by the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) [7].

RESOURCES

References:
[1] Kirkby, Lowry & Luongo, Francisco & Lee, Morgan & Nahum, Mor & Van Vleet, Thomas & Rao,
Vikram & Dawes, Heather & Chang, Edward & Sohal, Vikaas. (2018). An
Amygdala-Hippocampus Subnetwork that Encodes Variation in Human Mood. Cell. 175. 10.1016/j.cell.2018.10.005.
https://www.cell.com/cell/pdf/S0092-8674(18)31313-8.pdf
[2] Muse—Meditation Made Easy. (n.d.). Muse. Retrieved February 4, 2020, from
https://choosemuse.com
[3] Clutterbuck, J. (n.d.). Mind Monitor. Mind Monitor. Retrieved February 4, 2020, from
https://mindmonitor.com
[4] The Open Sound Control 1.0 Specification | opensoundcontrol.org. (n.d.). Retrieved February 4, 2020, from http://opensoundcontrol.org/spec-1_0
[5] Dunkels, A. (n.d.). Protothreads. Protothreads. Retrieved February 4, 2020, from                 http://dunkels.com/adam/pt/index.html
[6] devindelfino. (2019). Devindelfino/TelekinesisCar [Python].                                                                   https://github.com/devindelfino/TelekinesisCar
 (Original work published 2015)
[7] DelPreto, Joseph et al. “Plug-and-Play Supervisory Control Using Muscle and Brain Signals for Real-Time Gesture and Error Detection.” Robotics: Science and Systems(2018).
https://groups.csail.mit.edu/drl/wiki/images/d/d8/delpreto_rss2018_emg_eeg.pdf

Adafruit Industries | www.adafruit.com
InteraXon | www.choosemuse.com
Microchip Technology | www.microchip.com

PUBLISHED IN CIRCUIT CELLAR MAGAZINE • AUGUST 2020 #361 – Get a PDF of the issue

Sponsor this Article