Using Broadcom’s APDS-9960
The embedded technology available today for hand gesture recognition is truly impressive. Here, Jeff examines the capabilities of Broadcom’s APDS-9960 combo sensor. He investigates each function of the sensor—digital proximity, ambient light, RGB, and more.
— ADVERTISMENT—
—Advertise Here—
— ADVERTISMENT—
—Advertise Here—
While it’s been said Italians cannot talk if you tie their hands down, we all use hand signals every day. There’s the “thumbs-up” for yes, and “thumbs-down” for no. You probably know gestures that mean “stop,” “you,” “peace,” “love,” “one,” “two,” and some not-so-nice gestures. You may have played the game of Charades, where a player must act out a given word or phrase, without using any spoken clues. Often, a gesture will mean something different if motion is added to the hand sign. An umpire may use the “thumbs-up” hand signal to means “you’re out,” by moving his thumb up to his shoulder. Or a traffic cop might use the pointing gesture with a swing to the right or left as an indication of which way you should proceed.
Broadcom’s APDS-9960 is a combination digital proximity, ambient light, RGB and gesture sensor that can be used to detect many aspects of light. It incorporates both ambient/color light sensors and IR sensors. This article is about using each function of the sensor. If you refer to the block diagram of this tiny SMT part in Figure 1, you will notice that it contains an IR light source, but not a visual light source. By combining the IR transmitter and quad IR receivers, one can determine not only proximity, but also if there is any varying relationship between any of the quad IR sensors. Ambient light is necessary for the four RGB/light sensors—clear, red, green and blue. The device communicates over I2C, so we’ll need to connect it to a microcontroller to be able to use its magic. I’ll be using a Heltec HTIT-W8266 controller for its small size and 3.3V operation (Figure 2).
The block diagram of the APDS-9960 shows the IR transmitter, four IR receivers and four light sensors that make up the front end of this device. The system interface is I2C and gives access to a register set that includes specifics registers for the ALS Color, Proximity and Gesture engines.
Schematic of the APDS-9960. Our APDS-9960 module will share the same I2C bus as the display on the Heltec 8266 module. I have not implemented the display in this project; however, an RGB LED was added for demonstration purposes. You could use either connection format for the RGB LED, but I am using the common cathode connection in this project.
OPERATION
— ADVERTISMENT—
—Advertise Here—
The APDS-9960 likes 3V and 3.3V, but not 5V, so you’ll need some level conversion if you are not using a 3.3V micro. I have an APDS-9960 module, which takes care of bringing out the device connections from the SMT device to a 0.1” 6-pin header. The micro’s 8266 module uses its I2C bus for its onboard display, but it can be shared with the APDS-9960 module.
Once power is applied, the APDS-9960 does a bit of internal configuration and then goes directly to sleep! Any activity on the I2C bus wakes up the device, so it can service any communication that is directed to it. Because it’s such a light sleeper (ha), you can write and read to its registers at any time. Register 0x80 is of prime interest to the APDS-9960, because it has all the bits that identify what parts of this sensor will be active (Table 1). Separate bits enable each function that you wish to be active, the Gesture engine, the Proximity engine, the ALS Color engine, and the Wait engine. Proximity and ALS Color have additional bits that can activate an interrupt upon completion of that function.
While power is ON and at least one function is enabled, idle mode is entered—no more sleeping! All enabled functions are performed in round-robin fashion. We’ll take a look at these one at a time.
The APDS-9960 has many registers, but the most important might be the ENABLE register. You can see there are 4 bits here that enable individual functions, GEN, PEN, AEN and WEN. If bit0, PON, is disabled, the device will never even check to see if any of those four functions has been enabled by the user; it will merely go back to sleep.
PROXIMITY ENGINE
Proximity uses IR and time of flight to calculate distance. This sensor is designed for small distance. Maximum distance is about a decimeter, but this will depend on the register settings and reflective surface of the object within view. The IR LED and the four IR sensors are used for proximity (and gesture). The IR LED has several registers associated with its output. IR is emitted by configuring it with a pulse train of PPULSE pulses of some LLEN length (register 0x8E). The IR LED current LDRIVE can be set to one of four levels (register 0x8F).
On the receiving side, any or all four IR sensors can be used. Each can be enabled or disabled (register 0x9F). Since the outputs of the four IR sensors are essentially summed by pairs, if any are disabled, the sum will not be able to reach the same level as when all sensors are used. However, there are bits that can compensate for each disabled sensor (register 0x9F). There are four gain settings, PGAIN, that can be used to boost the signal up to eight times (register 0x8F).
When used for proximity, the Up sensor is paired with the Right sensor, and the Down sensor is paired with the Left sensor. The summed output of each pair produces PDATA. Each pair has compensation offsets that can used to match their outputs more closely when necessary POFSET_UR (register 0x9E) and POFSET_DL (register 0x9E). When the analog circuitry becomes saturated, possibly from too high a gain, PGSAT is set, and the result in PDATA should be questioned. PVALID is set when a new PDATA value is available. Both PVALID and PGSAT are in the STATUS register (register 0x9E).
Additional operations can be performed on PDATA. We can define low PILT (register 0x89) and high PIHT (register 0x8B) thresholds for PDATA. Only data falling between the two are recognized and increment the persistence counter; data falling outside will reset the persistence counter. When the persistence counter reaches PPERS (register 0x8C), PINT is set. If interrupts are enabled for proximity, then it is also set.
A note about interrupts here. Interrupts, as we are familiar with them, are not supported over the two-wire I2C protocol. You can poll the STATUS register to get that information. An additional output pin is provided by this device. It can be wired separately to an input pin on your micro, and provides a real external interrupt to your micro. The micro can then poll to find out what caused the interrupt. If your application is idle enough to handle polling, you can forego using a separate wired interrupt connection.
GESTURE ENGINE
For proximity, the APDS-9960 uses the four IR sensors in pairs and sums the output levels to determine if an object is reflecting the IR pulses that are being transmitted. In the Gesture engine, these sensors are used separately. The relative timing between sensors is used to determine any directional movement across the sensor field.
As previously mentioned, the engine is enabled with its own enable bit PEN (register 0x80). The gesture mode bit GMODE must also be set (register 0xAB) to enter the Gesture engine. This can be set by the user or when the PDATA (proximity) exceeds the gesture proximity entry threshold GPENTH (register 0xA0). The gesture dimension select register GDIMS determines which directional pairs will be used by the Gesture engine. The Gesture engine collects data sets for each enabled direction—Up/Down and Left/Right—and increments the gesture FIFO level count GFLVL (register 0xAA).
The gesture FIFO threshold in GFIFOTH (register 0xA2) determines when GVALID is set, along with GINT. It can be as little as 1 or as many as 16. A data set consists of a measurement from each of the four sensors. If GFLVL exceeds 32 (16 of each directional pair) then the FIFO has overflowed and the data sets should be tossed out.
When the gesture exit threshold level GEXTH (register 0xA1) is zero, this routine will exit; otherwise the data set is processed. Any value > GEXTH increments the persistence count, and any value < GEXTH clears the persistence count. As with the proximity, a persistence count (from 1 to 16) can be assigned to the Gesture engine. The persistence count is a required number of valid data sets that remain above the GEXTH threshold. Once the persistence count reaches the exit persistence count GEXPERS (register 0xA2), GVALID is set (and GINT).
It should be noted here that the GVALID and GINT could be set twice while the Gesture engine is operating. The first time, when GMODE=1, the FIFO has collected the required number of data sets. The second time, GMODE=0, the required number of those data sets is above the GEXTH. No decisions other than collection parameters are made internally. The user must ultimately determine what the data means.
There is also a wait state as part of the Gesture engine. The value in GWTIME (register 0xA3) can add a delay between data set collections. This can be set between 0 and 39.2ms. You may need this to delay each data set, depending on the speed of the hand gesture.
WAIT ENGINE
This is not so much an engine, but can be used to delay the amount of time between the use of IR and the ALS Color sensors. WTIME (register 0x83) can be set between 0 and 712ms. This can be extended by a factor of 12 times, by setting WLONG in the CONFIG1 register (0x8D).
ALS COLOR ENGINE
The last engine in this multi-function loop is the Ambient Light Sensor or Color sensors (Figure 3). Located diagonally between the orthogonal IR receivers, these sensors physically filter the light to which each sensor is exposed. There are receivers for red, green, blue and white. This engine is enabled with the AEN bit in the ENABLE register (0x80). 16-bit data is collected for each sensor. The gain for all sensors can be set from 1x to 64x via AGAIN in the CONTROL register (0x8F). Each time a sample has been taken, the AVALID bit is set.
Spectrum response of each sensor. The IR transmitter is separated from the eight sensors by a barrier that limits IR to only reflected energy. The sensor may be sealed behind a suitable cover if necessary, but make sure the material is transmissive to the frequency of interest.
Valid samples are further refined based on the clear sensors data, since it will be the sensor with the highest sample value. If a sample falls below the low threshold AILTH:L or above the high threshold AIHTH:L, then the persistence count is cleared. Only when the count exceeds the persistence count APERS (register 0x8C) will the routine exit with AINT set in the STATUS register (0x93).
While polling for AINT =1, you should also check the CPSAT bit, since this can indicate that the CDATA (clear) has become saturated. If CDATA is saturated, RDATA, GDATA, BDATA may also be suspect, so toss ‘em out.
ADDING AN RGB LED FOR VISUAL FEEDBACK
The APDS-9960 module does not contain any visible LEDs. Unless the user provides an external visible light source, the module uses available ambient light. Therefore, color information is only relevant when sufficient ambient light hits the sensor. I thought I would use an RGB LED as an indicator, and not a light source. By providing three PWM outputs from the micro, I could illuminate the RGB LED with any color and intensity I wanted. This device is a Bivar R50RGB-4-0045 4-pin DIP module, capable of high output levels. I’m using outputs 5, 6 and 9 as the drive pins to source the RGB LEDs with the 3.3V micro. The program allows the uses of common cathode RGB LEDs.
I use the analogWrite() command to set the PWM to any value from 0 (OFF) to 255 (ON) to adjust the brightness of each color. PWM values used to set our RGB LEDs are 8 bits and similar to the high byte of the color data registers RDATA, GDATA and BDATA from the ‘9960 color sensors.
Much of the initialization of the APDS-9960 is done for us during the self-initialization process. You need to set PON and AEN to enable the ALS Color engine. You also need to set AIEN if you want to use the physical interrupt. Since the APDS-9960 is an I2C device (address 0x39), you need to include the I2C library, wire.h (Listing 1). I’ll skip the constant and variable initialization and get right to the meat.
I2C ROUTINES
The first routine here displayID() attempts to read the ID register (0x92) (Listing 2). The value read back from the device should be 0xAB. This is a constant that identifies the device as an APDS-9960.
Listing 1
Only the I2C library is used in this Arduino sketch. In the setup() function, the serial port and the I2C communications are started, the RGB LED ripples through a few colors and communication with the APDS-9960 is checked.
#include <Wire.h>
void setup()
{
Serial.begin(115200); // start serial UART
analogWrite(PIN_WIRE_RED, 0); // start a PWM with the RED LED off
analogWrite(PIN_WIRE_GREEN, 0); // start a PWM with the GREEN LED off
analogWrite(PIN_WIRE_BLUE, 255); // start a PWM with the BLUE LED on
delay(500);
analogWrite(PIN_WIRE_GREEN, 255); // start a PWM with the GREEN LED on
analogWrite(PIN_WIRE_BLUE, 0); // start a PWM with the BLUE LED off
delay(500);
analogWrite(PIN_WIRE_RED 255); // start a PWM with the GREEN LED on
analogWrite(PIN_WIRE_GREEN, 0); // start a PWM with the BLUE LED off
delay(500);
analogWrite(PIN_WIRE_RED, 0); // start a PWM with the RED LED off
//
Serial.println(“ftb382 APDS-9960 Explorer”); // sign-on message
//
Wire.begin(); // start I2C
if(displayID() == 1) // is the APDS-9960 on the I2C bus
{ // no
Serial.println(“Error contacting APDS-9960”); // error message
while(1); // endless loop
}
else
{ // yes
Serial.println(“APDS-9960 OK”); // OK message
}
}Listing 2
Here are a few support routines for I2C in the setup() function in listing 1.
boolean readI2C()
{
Wire.beginTransmission(i2cAddress); // start bit
Wire.write(registerAddress); // device address
int error = Wire.endTransmission(); // stop bit
if(error) // any error
{ // yes
Serial.println(“Error:” + String(error)); // error message
return 1; // fail
}
else
{ // no
Wire.requestFrom(i2cAddress, 1); // start device address
registerData = Wire.read(); // read data
return 0; // success
}
}
//
boolean writeI2C()
{
Wire.beginTransmission(i2cAddress); // start bit
Wire.write(registerAddress); // device address
Wire.write(registerData); // device data
if(Wire.endTransmission()) // stop bit
{
return 1; // fail
}
return 0; // success
}
//
boolean displayID()
{
registerAddress = 0x92; // device address
if(readI2C())
{ // read fail
Serial.println(“I2C Error:” + String(registerData,HEX)); // error message
return 1; // fail
}
else
{ // read success
if(registerData == 0xab) // match
{ // yes
Serial.println(“ID = APDS-9960”); // match message
return 0; // success
}
else
{ // no
Serial.println(“ID = unknown”); // error message
return 1; // fail
}
}
}To read this register in the device, we need two additional functions, readI2C() and writeI2C(). Since this is a registered device, we must first tell it which register we are interested in (write) and then get or put data to that register (read or write). Note here that readI2C() actually contains two write commands to set the APDS-9960’s register. An actual writeI2C() routine contains not only those two write commands to set the APDS-9960’s register but also a write data command.
This is the basis for all communications with the APDS-9960. From here out, we only need to know what registers we want to read from (or write to). To enable the ALS Color engine, we set bit0 (PON=1) and set bit1 (AEN=1), write a value of 3, to the Enable register (0x80).
RegisterAddress = 0x80; // register of interest
registerData = 0x03; // data to write
writeI2C();
// I2C function
WHAT’S ON THE MENU
I’ve set up a few menus to allow the user to play around with the APDS-9960 module. The main menu allows the user to choose one of the three main functions.
Main Menu
Hit…
A: Color ALS registers
P: Proximity registers
G: Gesture registers
I use the Parallax Serial Terminal program on my PC to display output from this project, because its screen formatting options allow control characters to be applied—that is, 0x00 (clear the screen). This way, you can display updated values without having them scroll down and off the screen. Although you can use the Arduino IDE’s Serial Monitor, since these control characters aren’t recognized, things will scroll print fast.
When any of the three Engines are chosen, you will get a list of all the registers appropriate for that function.
In this Color ALS register sub menu in Listing 3, present values of all registers are presented in Hex and Decimal values. Refer to the APDS-9960 data sheet on how to interpret these registers. I’ll try and give some hints when it comes to writing to a register. If you type in an “A,” for instance, you get:
0b00000011
ENABLE = x-GEN-PIEN-AIEN-WEN-PEN-AEN-PON
Enter a value for register 0x80 (3)
the present value in binary, and a hint at what each bit of the ENABLE register represents. It will wait for a numeric entry, and revert back to the submenu. You can exit the submenu using “?” as input.
Because a register may affect multiple engines, like the ENABLE register 0x80 available in each submenu, you want to be careful what you’re writing, since you might be enabling more than one function at a time, and things can get tricky or be misinterpreted. So now that you have complete control over all registers, what’s next?
The main loop handles the menuing (Listing 4). If the MMChoice is X, then we are looking for the user to enter A, P or G. If one of these menu choices is entered, MMChoice becomes that entry, and a either a submenu is displayed, or MMChoice is forced to X and the main menu is displayed.
Listing 3
When the "Color ALS registers" is chosen from the main menu, you will get a sub menu display of all the pertinent registers associated with color ALS. The present value is also given for each register. Entering an appropriate sub menu character will allow you to modify the register value.
Color ALS registers
A - 0x80 = 0x3 (3) read and write
B - 0x81 = 0xc8 (200) read and write
C - 0x83 = 0xff (255) read and write
D - 0x84 = 0x4 (4) read and write
E - 0x85 = 0x35 (53) read and write
F - 0x86 = 0x0 (0) read and write
G - 0x87 = 0x0 (0) read and write
H - 0x8d = 0x60 (96) read and write
I - 0x8f = 0x0 (0) read and write
J - 0x92 = 0xab (171) read only
K - 0x93 = 0x11 (17) read only
L - 0x94 = 0x98 (152) read only
M - 0x95 = 0x0 (0) read only
N - 0x96 = 0x25 (37) read only
O - 0x97 = 0x0 (0) read only
P - 0x98 = 0x2d (45) read only
Q - 0x99 = 0x0 (0) read only
R - 0x9a = 0x47 (71) read only
S - 0x9b = 0x0 (0) read only
T - 0xe4 (0) write Only
U - 0xe6 (0) write Only
V - 0xe7 (0) write Only
? - back to Main MenuListing 4
The loop() function handles displaying the main menu or diverting program flow when a sub menu is selected. The showMainMenu() function will also check for enabled sensors.
void loop()
{
if(MMChoice == ‘X’)
{
if(Serial.available())
{
myValue = Serial.read();
MMChoice = myValue;
}
}
switch(MMChoice)
{
case’A’:
lastMMChoice = ‘A’;
showSubMenuALSColor();
getChar();
if(myValue>0)
{
SMChoice = myValue;
doALSColorChoice();
}
break;
case’P’:
lastMMChoice = ‘P’;
showSubMenuProximity();
getChar();
if(myValue>0)
{
SMChoice = myValue;
doProximityChoice();
}
break;
case’G’:
lastMMChoice = ‘G’;
showSubMenuGesture();
getChar();
if(myValue>0)
{
SMChoice = myValue;
doGestureChoice();
}
break;
default:
MMChoice = ‘X’;
showMainMenu();
break;
}
delay(100);
}Note that if no user entry is made, the main menu is displayed again and the output handle for any engine that is enabled. Two bits must be set for anything to happen. First, the power-on bit (PON bit0) of the ENABLE register must be set. Second, one of the engine enables, AEN, PEN or GEN, must be set to enable a function. Since any menu or submenu, begins with a clear screen, we will be able to show what’s going on with each function in real time. As part of the main showMenuMenu() function, this code will check the status of each engine, based on the last Main Menu choice, lastMMChoice.
switch(lastMMChoice)
{
case ‘A’:
look4Color();
break;
case ‘P’:
look4Proximity();
break;
case ‘G’:
look4Gesture();
break;
default:
break;
}
Each of these three functions, look4Color(), look4Proximity(), and look4Gesture(), will check to see if PON is set and if its particular function is set. If these are satisfied, the status register is checked. Like the ENABLE register, the STATUS register is important (Table 2).
To get insight into the device, read the STATUS register. Two bits here indicated you may want to disregard any reading, since the register values used produced saturation of a sensor. Three bits show if an engine produced a physical interrupt output. The final two bits show when valid data is ready for you to read. Note that the GVALID bit can be found in the GSTATUS register (0xAF).
IT’S ALL IN THE INTERPRETATION
Ever been to an art gallery? It doesn’t matter whether it’s paintings, sculptures, music or photography; the beauty (or disgust) is in the eye of the beholder. We all seem to have a different slant on what we see, hear or think. The spectral diagram of the sensors used in this device is typical (Figure 3). You will note that the spectra of the individual sensors overlap one another. This shows that a red sensor will see green, just not at the same intensity as green sensor. While this isn’t exactly what we would like or expect, we need some interpretation to determine if this misrepresented sensor output is important.
For the ALS Color engine the C, R, G and B sensors are used to determine which component has the strongest signal strength. Upon finding AVALID set in a STATUS register read, the four Color data register pairs, CDATAH:L, RDATAH:L, GDATAH:L and BDATAH:L, are read (must read LSB to latch MSB). Values are reported via the serial port. The color with the highest value is chosen to illuminate the corresponding color RGB LED. All reporting is done while at the main menu level.
For the Proximity engine, the four IR sensors are paired U/L and D/R, with their outputs combined to determine the strength of any reflected IR. Upon finding PVALID set in a STATUS register read, enabled IR sensors are combined, producing a single value, PDATA. Using the RGB LED’s red LED, PDATA is used to set the intensity of the red LED.
As you approach the APDS-9960 with your hand, you will affect the red LED’s intensity. With the default settings, proximity will begin at about 2”, and the intensity of the red LED will increase up to a maximum at the sensor. To demonstrate a bit of control, I’ve defined three constants. Any value less than MinClearCount will not be processed. The same value must be seen MaxProxCount times before the red LED is changed. New values must be within ProximityHysteresis of the last value for it to be recognized as the same. This allows the red LED’s intensity to be held until any new value has passed muster.
#define ProximityHysteresis 10
#define MaxProxCount 3
#define MinClearCount 5
For the Gesture engine, the IR sensors U, D, L and R are used to gather any reflected IR. Upon finding GVALID set in a GSTATUS register read, you may discover how many samples have been placed in the FIFO stack by reading the GFLVL register. These sets of data can be read from the four GFIFO registers, GFIFO_U, GFIFO_D, GFIFO_L and GFIFO_R. If all timing, how the registers are set and the speed of the hand are optimal for a hand waving left to right, you might expect a scenario like this. Assuming the hand is approaching along the L/R axis. A reflection is first picked up by the right sensor, then the up and down sensors, and finally the left sensor. When the hand leaves the sensors, the reflection is lost by the right sensor first, then the up and down sensors, and finally the left sensor. Note while all sensors might see a rising and falling of reflectivity, it is the timing of the edges that suggest direction.
CONCLUSION
Getting so much information from a tiny sensor can be overwhelming (Figure 4). Using the ALS Color engine and the Proximity and Gesture engines together may cause some crosstalk between the IR and visual sensors to skew readings, so keep those operations separate. Also, I would like the device to have a visual LED light source, so the ALS Color engine could be used as a reflective device, without having to depend on ambient light from the rear to get around the physical module.
I used a Heltec HTIT-W8266 micro for its small form factor and 3.3V operation. Although I did not use the onboard display, the format is very handy and compact. Since the APDS-9960 is a tiny SMT device, using one of the modules available with the device made it all work nicely. I added an RGB LED run by the micro to give the user feedback on the sensor’s functions. Otherwise, the USB not only powers the whole circuit but also serves as a debugging tool.
Now that you know how the APDS-9960 module works, you can use the libraries available for it. Although these make the module easy to use, depending on the library you are using, you may be limited in what functionality is supported. You can probably see how this might be used to design a touchless switch, for those high-traffic areas. We are in a new age now, like it or not. I never thought we‘d be considering the spread of COVID-19 a safety issue. Still learning after all these years.
Additional materials from the author are available at:
www.circuitcellar.com/article-materials
RESOURCE
APDS-9960 – Digital Proximity, Ambient Light, RGB and Gesture Sensor
Broadcom, www.broadcom.com
PUBLISHED IN CIRCUIT CELLAR MAGAZINE • MAY 2022 #382 – Get a PDF of the issue
Sponsor this Article
Jeff Bachiochi (pronounced BAH-key-AH-key) has been writing for Circuit Cellar since 1988. His background includes product design and manufacturing. You can reach him at: jeff.bachiochi@imaginethatnow.com or at: www.imaginethatnow.com.
