Build an Automated Fruit-Ripeness Detector

Banana Scan

Embedded systems can be tasked to do many things that are too tedious for humans. Watching fruit ripen is a good example. In this article, learn how these three Cornell students built a device that uses spectroscopy to track the ripeness of a variety of fruit, including bananas and oranges. The system uses a TFT display and a Microchip PIC32 MCU.

FRUIT SPECTROSCOPY 101

As fruits ripen, their surface colors change, due to a decrease in the concentration of chlorophyll-a [1]. This suggests that color changes in fruits could potentially be a good proxy for measuring their ripeness. The reflectance of light waves at 678nm is inversely related to the concentration of chlorophyll-a [1]. These findings are summarized in Table 1.

Red light has a wavelength of 678nm. Chlorophyll-a absorbs red light and reflects green light, so it is not surprising that as the concentration of chlorophyll-a decreases, the reflectance of light at 678nm increases. To measure the intensity of the 678nm light reflected off fruit to determine their ripeness, we obtained a reasonably priced ($26) SparkFun NIR spectral sensor AS7263 board. The sensor module has six photodiode channels, including one with peak responsivity at 680nm. In addition, the sensor has a full-width, half-maximum bandwidth of 20nm, an integrated LED for light emission, and UART capability. This device was driven by a Microchip Technology PIC32 MCU.

We agreed that certain criteria must be met, in order for a fruit ripening detector to be ready for further commercial or industrial development. The user must be able to differentiate if regions of a scanning area are interpreted as ripe or unripe on a display. The displayed levels of ripeness must be computed from measurements taken on a spectral sensor that has demonstrated quantitative efficacy. Also, the detector must be automated, which means that any movement of a spectral sensor must be accurate, so that coordinates on the display match with sensor progression.

XY PLOTTER

There were several possible solutions to the problem of motorizing the spectrometer, such that it could scan under an area of fruit, while maintaining a constant distance from the surface of the fruit. The simplest option involved an XY plotter and an overhead acrylic panel. We salvaged a vintage XY plotter that included stepper motors and an Adafruit DC and Stepper Motor HAT (driver) for a Raspberry Pi (RPi) that was left over from an older project.

Adafruit provides a Python library for interfacing the Adafruit Motor HAT to control DC motors with speed control, and stepper motors with single, double, interleave and microstepping step styles. This library [2] was used in our final RPi control program. We always used double style for maximum speed in our program, with a motor rotational setting of 20 steps per revolution and a speed of 60rpm to move the sensor in 1cm increments. The spectral sensor transmitted the reflectance readings to the PIC32 after each centimeter movement.

We developed our own parallel communication scheme to synchronize sensor movement on the RPi end with the PIC32 sensor reading and the drawing of the heatmap. The protocol relied on four GPIO pins—three inputs from PIC32 to RPI, and one output from RPi to PIC32. Of the three inputs, two control bits determine the direction of the sensor movement and one ready bit from the PIC32 informs the RPi that the control bits are ready to be read. The output bit is for the RPi to inform the PIC32 that it has finished moving the sensor.

The program begins by polling for the ready signal from the PIC32. Once it receives the ready bit from the PIC32, it interprets the control bits: 00 for moving one step up, 01 for moving one step down, 10 for moving one step left, and 11 for moving one step right. Then it moves the stepper one step in the appropriate direction. After it has finished moving, it sets the acknowledgment bit high and begins polling for the ready bit to go low before starting from the beginning again.

The MLABX/XC32 environment from Microchip was used to program the PIC32. We used the Protothread library developed by Adam Dunkels [3], with extensions and macros developed by Prof. Bruce Land [4] at Cornell University. Our PIC32 software comprised four separate components: the reset functionality, scanning procedure, TFT color mapping and interfacing with the Raspberry Pi.

RESET FUNCTIONALITY

The movement of the XY plotter is controlled by either reset or scanning instructions. In the scanning procedure, the XY plotter moves the spectral sensor in a grid-like pattern to generate a color map with ripeness levels across the scanning area. Before conducting this scanning procedure, the XY plotter localizes the spectral sensor to a starting position according to reset functionality. The methodology of our reset functionality and scanning procedure was organized in the state machine diagram shown in Figure 1.

Our reset functionality required placing a thin strip of blue paper and a small area of pink paper on the left side and top-left corner of the acrylic sheet, respectively. They enabled the sensor to localize its position by discerning if it was under the blue sheet, the pink sheet or neither. If the sensor is not initially located beneath the blue sheet or the pink sheet, the sensor begins moving left until the blue sheet is detected (state 1). When the sensor reaches the blue sheet, the sensor begins moving up the blue sheet until the pink sheet is detected (state 2). The pink sheet is located directly over the reset position, so when the sensor reaches this area, it finishes the reset procedure by moving right until the scanning area has been reached (state 3).

Before each movement, the PIC32 would read 610nm and 680nm wavelength channels from the spectral sensor, to determine where the sensor was located beneath the scanning area. A UART connection between the PIC32 and the spectral sensor was utilized to obtain spectral measurements in µW/cm². Non-blocking threads were spawned to initiate the UART protocol and send commands to receive the spectral data from the sensor.

The PIC32 sent two UART commands to the sensor. ATLED1 turned on the integrated LED on the spectrometer, and ATCDATA returned comma-separated calibrated data from 610, 680, 730, 760, 810 and 860nm wavelength reflectances recorded on the sensor. The delimited data received from the ATCDATA command were separated into a six-element array that held spectrometer readings for each reflectance reading. When sending commands, we found that it was necessary to include a carriage return at the end of the command (for example, “ATCDATA\r“), to act as a termination character signifying the end of the UART transmission.

SCANNING PROCEDURE

After the reset procedure was completed, the program proceeded to the scanning phase. The scanning phase first consisted of stepping right a set number of times, based on the length of the acrylic sheet and the distance the sensor traveled at each step (state 4). Then the sensor would move one step down (state 6) and move left the same number of steps as it did moving right (state 5). Because the blue sheet was located to the left of the square acrylic, the scanning area had a bit more vertical distance than horizontal distance to cover.

In the end, the sensor moved 18 steps right/left and 20 steps down, to move through the entire scanning area. The program would keep track of the number of steps it moved right, left and down in variables called steps_right, steps_left, and total_steps_down, respectively. Steps_right was incremented while moving right in state 4, and was cleared to 0 once the sensor made a down movement in state 6. Steps_left was incremented while moving left in state 5, and was cleared to 0 once the sensor made a down movement in state 6. Total_steps_down was not cleared to zero until the spectral sensor had completed the scanning procedure.

We used a TFT LCD to display ripeness levels across the scanning area to the user. To draw graphics on the TFT display, we used the library ported to PIC32 from Arduino by Syed Tahmid Mahbub [5]. The TFT LCD was updated after each movement of the scanning procedure. After each movement, a region/block of the TFT display corresponding to the sensor’s previous position was colored according the level of ripeness detected there.

At the end of the scanning procedure, the generated color map consisted of 12×12-pixel blocks, with each block representing a 1cm² portion of the scanning area. A tft_fillRect() function with arguments for the top-left position of each block, the block width/height and the block color was used to draw each block. The pixel location of the top-left corner of each block was altered according to the current state of scanning phase. For example, the X coordinate of the top-left position was incremented by 12 pixels each time the sensor moved a step to the right:

//Fill a block on the color map for each movement
//Cursor_x and cursor_y represent the top left pixel of the block
//Third and fourth arguments are block width/height
//Fifth argument encodes the RGB color of the block
tft_fillRect(cursor_x, cursor_y, 12, 12, color_reading);
//Sensor moved right, so next block is right of the previous cursor_x+=12;

The color of each block was determined by mapping the spectral intensity readings of the 680nm wavelength channel into a color format encoding specified by the TFT display. The TFT uses an 11-bit color format, with the top 5 bits encoding red intensity, the middle 6 bits encoding green intensity and the low 5 bits encoding blue intensity.

Since the color map used only blue and red (blue to specify that no fruits were detected, and red to specify the detection of ripe fruit), the 0-512 intensity reading from channel 2 (680nm) was converted into a range of 0-31 to encode the 5-bit settings for red and blue. The final color gradient was computed by using the equations shown in Table 2.

The red value was set to whatever the converted intensity reading was, the green value was unused and the blue setting was modeled to be inversely related to the red value. The blue intensity was set to have a maximum value of 13 to improve the appearance of the color map (no bright blue or bright purple colored blocks). In this color-coding system, blue regions of the color map correspond to areas with no fruit, dark red regions correspond to areas with unripe fruit and bright red regions correspond to areas with ripe fruit.

USER INTERFACE

We designed an interface for the user to select a fruit to inspect for ripening. A scroll control potentiometer was hooked to an integrated analog-to-digital converter (ADC) on the PIC32 to select a fruit setting, and an external button was used to load the setting. The options were laid out on the TFT screen, and a white background highlighted the text that the user currently selected. The reset/scanning procedure would not begin until the user selected a type of fruit to scan in the user interface.

The three fruit currently on the interface are bananas, apples and avocados. Although we had these options for the user to choose, we did not implement separate functionality for each of them. A further improvement to this system would be to implement different thresholds for 680nm intensity for different fruit by changing the mapping of the color map. For example, we observed that avocados have a much lower range of values for 680nm intensity than bananas, for instance. This phenomenon is discussed in further detail in the Conclusions section.

To verify the qualitative functionality of the fruit ripeness detector, tests were performed by placing ripe and unripe fruit on the acrylic sheet. Our system accurately represented regions where ripe fruit, unripe fruit and no fruit were placed, after an entire sensor progression of the scanning area (Figure 2).

To demonstrate the quantitative efficacy of the spectral sensor, we took three sensor readings of two bananas per day over the course of 6 days, averaged the sensor readings, and calculated each day’s percent change from the day 1 reading. The results are summarized in Table 3.

They show a clear progression of increased reflectance in 680nm, which agrees with the reflectance results of the paper by Meng Li et al.—link available at [1]. Banana 1 reached a stable level of ripeness after the day 4. Therefore, the sensor reading stopped increasing. Photos of the bananas are shown in Figure 3. We measured the area between the two black lines for consistency.

CONCLUSIONS

Our project satisfied our criteria for an adequate fruit-ripening detector. The user could differentiate areas with unripe bananas and oranges from those with ripe bananas and oranges (Figure 2). Also, the system sacrificed speed for accuracy; scans of the entire 25.40cm × 30.48cm scanning area took around 15 minutes to complete. A full schematic of the MCUs and connections is given in Figure 4.

Performance with different fruit: Although our system worked well with bananas and oranges, ripeness levels on the display were not as discernible for avocados, with gradients of dark purple for both semi-ripe and unripe avocados. This is because avocados are dark green throughout their ripening process (until they become black after ripening for a long time). Since chlorophyll-a and the skin of the avocado share similarities in their peak wavelength absorption, the range of intensity values for unripe and ripe avocados are lower than the range of 680nm intensity values for unripe and ripe avocados is lower than the range of 680nm intensity values for unripe and ripe bananas/oranges.

All that said, we could have adjusted the color map according to the fruit specified by the user, to interpret a lower range of values for dark green fruit such as avocados. This was our original plan for the final design, and we coded the user interface for this very purpose. But under this design, only one fruit type could be scanned at a time, unless additional thresholding was used to identify the fruit type before ripening.

Reset procedure: Another feature that could be improved in further development is the reset procedure of the system. It did not work as expected at all times, with the sensor sometimes failing to detect the blue paper mounted on to the acrylic. We realized that the reset functionality of our system was inconsistent, because the intensity values of the 610nm and 680nm channels are more irregular for colors (such as blue or pink) that do not have peak absorbance/peak intensity at those wavelengths. A more suitable mechanism for resetting the system would be to use limit switches. The limit switch implementation would offer a more consistent reset procedure, by detecting when vertical or horizontal walls were hit by the sensor. 

RESOURCES

References:
[1] Li, Meng, et al. 1997. “Optical Chlorophyll Sensing System for Banana Ripening.” Postharvest Biology and Technology, 12(3):273–283. doi:10.1016/s0925-5214(97)00059-8.
https://www.semanticscholar.org/paper/Optical-chlorophyll-sensing-system-for-banana-Li-Slaughter/3e3179c6a92077c9b7c9e43a712bb4cdf1a17545
[2] https://github.com/adafruit/Adafruit-Motor-HAT-Python-Library
[3]http://dunkels.com/adam/pt
[4] http://people.ece.cornell.edu/land/courses/ece4760/PIC32/index_Protothreads.html
[5] https://people.ece.cornell.edu/land/courses/ece4760/PIC32/index_TFT_display.html

Adafruit | www.adafruit.com
Microchip Technology | www.microchip.com
Sparkfun | www.sparkfun.com

PUBLISHED IN CIRCUIT CELLAR MAGAZINE • JANUARY 2021 #366 – Get a PDF of the issue

Christina Chang, Michelle Feng and Russell Silva

+ posts

Christina Chang graduated from Cornell University with a B.S. in Electrical and Computer Engineering and a M.Eng. in Computer Science. She currently works for Movandi, a 5G startup, as a systems engineer. She likes to garden and cook in her free time.

Michelle Feng graduated from Cornell University with a B.S. in Electrical and Computer Engineering. She currently works for Leidos as a software engineer. She likes to volunteer and play video games in her free time.

Russell Silva graduated from Cornell University with a B.S. in Electrical and Computer Engineering. He now works for Viasat on airborne radio-frequency terminals which provide satellite internet in-flight. He likes to produce music in his free time.

This author does not have any more posts.