Using Computer Vision and RPi
Doing any kind of real-time machine vision once required expensive, complex computing systems. Today, low-cost, widely available resources have changed that game. Demonstrating just that, these Cornell engineering students used Raspberry Pi hardware, Python programming and a robotic arm to make the perfect s’more.
For any of you readers unfamiliar with the term, a “s’more” is a toasted marshmallow and a piece of chocolate sandwiched between two graham crackers. With that in mind, we named our project: SMORE, which stands for S’more Mechanism for Optimal Roasting Experience. This project was inspired by our mutual love of camping and the unique challenge of getting a marshmallow toasted to perfection. With SMORE, a user can simply input a desired level of “toastiness,” and—using robotics and computer vision—SMORE takes care of the rest using a Raspberry Pi [1].
The-high level design of SMORE is quite simple (Figure 1). One motor controls the alignment of the marshmallow in front of the heating element. The camera situated above the marshmallow keeps a close eye on the color as the marshmallow browns, attempting to detect the color that the user selected at the start of execution.
Once the toasting is complete, the heating element turns off, and a claw packed with a graham cracker—plus an optional piece of chocolate—moves forward on a rack and pinion to acquire the marshmallow and return it for the user to remove and consume.
HARDWARE
The mechatronic aspects of this project fall into four major categories: the roasting apparatus, the video monitoring, user input and the s’more assembly. The components are contained in a laser cut, wooden chassis to make it portable and protect the assembly from interference. The primary hardware aspect of the device is the marshmallow-toasting capability. We obtained the heating element from an old toaster, which we disassembled to acquire the mica and metal heating element from within.
The heating element runs off wall outlet power, which we switched via a 10A 120V relay, controlled by the Raspberry Pi. The heating element is elevated from a wooden platform, using metal standoffs to isolate it from any flammable materials. The entire wooden chassis is insulated with fiberglass covered in aluminum foil, so as not to radiate excessive heat. The marshmallow sits on a two-pronged dowel attached to a continuous rotation servo [2], which slowly rotates the marshmallow to allow it to roast in a controlled fashion.
The toasting of the marshmallow is monitored by a Raspberry Pi Cam V2 [3], which is mounted above the roasting apparatus. The Pi Cam is situated in a plastic case to protect it from both the heat and any particles of marshmallow or fiberglass. The user gives input and receives feedback on how toasted a marshmallow will turn out, by means of a selection menu for “roastedness” level operated with buttons.
— ADVERTISMENT—
—Advertise Here—
The most challenging mechanical aspect of SMORE was the claw assembly component. Starting out with an example mechanical claw [4], the team modified the designs to account for gripping a larger item and to have automatic actuation via servo. The gripping arms of the claw had to hold a graham cracker without dropping or crushing it, support chocolate or other optional toppings and drag the marshmallow off the prong and onto the graham cracker plate. The claw also had to allow a user to easily remove the assembled s’more after roasting. To accomplish this, we redesigned and 3D-printed the claw to have a graham-cracker-sized crevice into which the components of the s’more were inserted and easily removed (Figure 2).
We also modified the mounting end of the claw to hold a servo, such that it could be automatically opened and closed on command when the marshmallow had completely toasted. After the claw was operating properly, we designed the rack and pinion to carry the claw to the marshmallow and remove it from the prong. To do this, we laser cut a mount to attach the claw and servo, which sits on a sliding panel. The panel is pushed back and forth by linear teeth interfacing with a gear attached to a stepper motor [5], which pushes the platform. The claw-panel assembly is kept in place by the rack on which the panel slides. The entire rack-and-pinion holder system was designed to be laser cut for easy prototyping and iterating. Figure 3 shows the CAD drawing of the complete system.
SOFTWARE
The software component for SMORE has three main phases of operation: Pre-Toasting Setup, Toasting and Retrieval. The Pre-Toast programs detect and identify the marshmallow (Figure 4), and take user input for toasting levels (Figure 5 and Figure 6). During Toasting, one program monitors the marshmallow (Figure 7), controls the heating element and provides some additional safety layers. The Retrieval program drives several motors to extend the retrieval claw, close it and return with the s’more. After these phases, a Wrapper Script ensures that all the programs work together sequentially. More on the Wrapper Script later. A collection of all programs and other related files are available on Circuit Cellar’s article code and files webpage.
user_in.py. The options on the right side correspond to buttons the user interacts with to cycle which level of darkness is desired.
user_in.py GUI after the user has cycled to a desired darkness level of four and pressed the button corresponding to “select” on the previous GUI screen. Now the user can either confirm the choice or cancel to select a different level.
rt_one_side.py. The most recently read darkness value of the marshmallow is shown in the upper left corner of the screen. Image blur in this instance is due to camera warping. (Right) Example of the concurrent terminal readout during operation, displaying the initially read darkness level, the amount of darkening to be done based on the user input and the target value.Pre-Toasting Phase: Before beginning the roasting process, the user manually loads a marshmallow. Due to potential deviation in the placement and shape of each individual marshmallow, we wrote a program to detect contours in the video feed that corresponded to a marshmallow. The specific parameters assessed are the perimeter of the contour and the brightness of the enclosed area. The program then awaits user confirmation of a contour corresponding to the marshmallow, before passing said contour back for other programs to use. A full discussion of this pick_countour.py program is given below.
After the marshmallow has been correctly identified, the user is prompted to select a level of toastiness for the operation of the device. The interface for this selection menu leveraged a simple GUI and buttons connected to the Raspberry Pi to cycle through options for the level of darkness desired. More on this program, called user_in.py later.
The program written for pick_contour.py begins with analysis of the live video feed from the Pi Camera using the OpenCV [6] Python library, cv2. Each frame is first cropped to focus on the segment of marshmallow skewer that could potentially have a marshmallow on it. After the cropping, the image is moved into the analysis stage of the program. The thresholding process is straightforward with cv2 supported functions, since a grayscale version of the frame can be made with one function call, and then a threshold applied with another function. These functions are: cvtColor to change to grayscale and threshold to threshold the image.
After thresholding, the image is analyzed for contours using the findContours function, which determines if the specific contour could potentially be a marshmallow. Following several tests with different attributes, we found that the most reliable and simple solution to the problem was to validate a contour based on its perimeter and average pixel color.
Because the image had been focused on only the potential marshmallow area, the contour corresponding to it is significantly longer than anything else in the image, and thus narrowed the image down to a maximum of two contours each time tested, once finding an appropriate cut-off level. The process of narrowing down contours by pixel darkness values was handled in a subtle way during thresholding. To assure proper alignment during contour finding, and for consistency later in measurement, the servo controlling the marshmallow skewer is also set to an initial position before identifying the contour, thus creating a consistent reference location.
The program begins with an initial wait time of 3 seconds before it actively searches for the marshmallow, to allow the Pi Cam to adjust to light settings and refocus. Then the program allows the user to select which contour is correct, if several were identified. Additionally, the verification step provides additional safety, so the device doesn’t start without user confirmation. Verification is handled by stepping through each potential marshmallow contour for a given image, until one of two conditions is met. If the user confirms the contour as a marshmallow, then this contour is recorded and the program ends. However, if none of the contours in the frame is selected, the next frame is loaded and analyzed in the same sequence. Figure 4 shows the display screen with the contour outlined in blue. The cause of image blur is discussed in the ISSUES AND SOLUTIONS section later in this article.
— ADVERTISMENT—
—Advertise Here—
The user_in.py program uses four buttons to allow a user to navigate a GUI to select a toasting level. The display in its initial stage (Figure 5) contains a generic s’more image and user options: Select – which selects the current level of toastiness displayed as the toast level; Lighter – which decreases the level of toastiness by one stage; Darker – which increases the level of toastiness by one stage; and Quit – which quits the program and execution of the SMORE software.
After the user cycles to a desired toastiness level (1-10) and selects the option, a second confirmation page is displayed. The page prompts the user if the level of toastiness is correct, and gives two options (Figure 6): Confirm, which confirms the level; or Cancel, which deselects and allows the user to choose a different value. Once the user has confirmed a desired toast level, the information is recorded and sent back for use later before the program ends.
Toasting Phase: Once Pre-Toasting is finished, the software flow of SMORE toasts and monitors the marshmallow. Our original objective was to roast the entire circumference of the marshmallow, but after several operational issues (discussed later under ISSUES AND SOLUTIONS), functionality became limited to analysis and roasting of only one particular side of a marshmallow. In this slightly reduced execution, the main elements were: ensure that a consistent side is aligned for the toasting and measuring operations via a servo; analyze only the marshmallow contour for toastiness level; determine when the marshmallow has reached the desired toast level; control the heating element; and provide some interrupt ability for the user.
The first step in development of the rt_one_side.py program was averaging pixel values for a region targeted by the previously defined contour. To accomplish this, a function was created that took in a grayscale image and masked the pixels corresponding to the area inside the contour. The average of the corresponding pixels in the grayscale image is returned as the toastiness level. The grayscale image is obtained the same way as in pick_contour, where simple cv2 function calls could capture a video feed from the camera and analyze frame by frame.
After a region was analyzed completely, the next step was to control the servo motor. Every 10 seconds during operation, the marshmallow was rotated from optimal toasting position (+90 degrees) to the best analysis position (0 degrees), analyzed and returned to the toasting position. To achieve consistent alignment, a standard servo moves the marshmallow to each position.
Once the monitoring and measurement aspects were complete, the next logic to tackle was how the toasting process would be determined as complete, and to enable the user to manually end the toasting process. To evaluate the toast level of the marshmallow, an initial value is measured to determine the base darkness of the marshmallow. Then a target value is calculated subtracting 5 × toast_level from the base darkness level, where toast_level is the desired value passed in from the user_in script. Upon reaching the calculated toast value, the rt_one_side script completes execution.
As a safety measure, should anything go awry in the toasting process, the user can press a button that forces the toasting program to complete and begin retrieval of the marshmallow. For each toast level read, an average of five readings is taken to help mitigate any issues from deviations in frames that could prematurely trigger a retrieval. The start value is also read after several seconds of operation, to allow the camera feed to stabilize after the heating element is turned on.
The final aspect of control is manipulating the heating element. The program switches the heating element ON via a relay at the start of operation, and OFF at the end. During operation, a live camera feed of the marshmallow is displayed to the user to monitor toasting progress. The feed also displays the last read toastiness level on the image in the upper left corner. Figure 7 is an example of the feed during operation. For additional tracking of progress, the program also prints to the terminal the read starting toast level, the level of toasting to be done (cook level), and the target toast level.
Retrieval Phase: The final stage of execution for SMORE is to physically retrieve the toasted marshmallow. The software to support the program is contained in one program, retrieve.py. The retrieval components are the continuous rotation servo that opens and closes the claw mechanism, and the stepper motor that drives the rack and pinion to move the claw laterally to the marshmallow.
The first stage of retrieve.py execution is to ensure that the claw mechanism is open, so the marshmallow can fit inside the jaws. To do this, the servo controlling the claw is driven counterclockwise for approximately 1.5 seconds. After the delay, the claw mechanism is driven to the marshmallow via the stepper motor in the rack-and-pinion system. After several trials, stepping full steps at a frequency of 100Hz for 6 seconds was found to be a satisfactory set of parameters for this operation.
Following this, the marshmallow is aligned inside the jaws of the claw. Then, the jaws can be closed by driving the servo clockwise for 1.5 seconds again. Finally, the claw mechanism is retracted by driving the rack-and-pinion stepper in the opposite direction for 6 seconds, by inverting the value of the DIR pin on the stepper motor driver [7]. Once these stages have completed, the s’more is removed from the device.
Wrapper Script: All the programs discussed so far were developed as independent modules, rather than as one massive program handling everything. However, it was essential for them to work together sequentially. To accomplish this, a wrapper script, op.py, was developed. In the op.py script, each supporting Python program was imported and executed as its own function call. The first program executed was pick_contour.py, which returned the contour to be recorded in the wrapper script as variable C. The next program was user_in, which stored the user selected toast level in the variable TL. When calling rt_one_side, each of the aforementioned saved variables is passed in, so that the program knows the desired toast level and the target contour.
After rt_one_side completed, a 5 second delay ensures enough time for cool-down of the heating element, to prevent the claw from melting as it grabs the marshmallow. Following the delay, the retrieve program executes to fetch the marshmallow and build the s’more simultaneously. After retireve.py is completed, the wrapper script completes. Figure 8 below is a video of the full operation of these scripts and the hardware they control.
FIGURE 8 – To truly appreciate the mechanical design of SMORE, watch this video of it operating. (Original video at https://www.youtube.com/watch?v=g7wXXy9KvRw&feature=emb_logo)
ISSUES AND SOLUTIONS
We encountered several bumps in our road to developing SMORE, including issues with the heating elements and camera. The heating elements from the toaster provided several challenges. There were two types of heating elements—one single-wrapped side with less resistance, and one double-wrapped side with more resistance. The single-wrapped elements generated so much heat that the marshmallow began to burn quickly after presenting the first signs of toasting, making quick detection almost impossible.
— ADVERTISMENT—
—Advertise Here—
The double-sided element was a better option, but was a bit too far to the other extreme. It generated enough heat to melt the marshmallow, but wasn’t effective at toasting it rapidly. The marshmallow melted too much to rotate on the skewer, and instead, the skewer rotated inside the stationary marshmallow. To overcome this, only one side of the marshmallow was toasted, and a double pronged skewer was developed to prolong the effective rotation duration.
The heating elements also created a problem with the camera. The fully contained element caused the internals to become quite hot, and melted a part of the camera after multiple trials. This caused the display to become blurred (Figure 4).
The next problem was in the program pick_contour, where the contour found for the marshmallow was incorrect. The issue stemmed from three different factors. First, the bottom of the heating chamber was somewhat reflective and was detected as a part of the marshmallow contour. Second, residue from previous marshmallows along the skewer read as being a part of the present marshmallow. Third, the melting of the camera from previous runs created blurred images that further skewed the contour. These issues were managed by painting the bottom of the platform black to provide better contrast, wiping off the rods between runs, and not running the apparatus for long periods of time.
The remaining issues occurred with the rt_one_side portion of software execution. First, the system read much smaller initial toast values on the first several reads, compared to subsequent measurements. This was likely due to dynamic lighting settings as the heating element turned on, and was partially solved by setting the “start toast” level after the heating element had fully turned on and started to glow.
The second problem during the program function was that the toastiness of the marshmallow rapidly increased in the final stages of toasting. This made it difficult to reach the desired levels of toastiness without the marshmallow catching fire. We partially solved this issue by increasing read times as the marshmallow neared more toasted values.
RESULTS AND CONCLUSIONS
In the end, SMORE produced many delicious s’mores. If we’d had more time to develop the machine (Figure 9), we would have had better lighting, protected the camera better and perhaps added more redundant measures to protect the heating element (such as adding a fuse, and enlarging the chassis).
The final function of the SMORE system was to detect a marshmallow, take user input for a desired toastiness level, attempt to toast one face of the marshmallow to a given darkness level, retrieve the marshmallow from the heating chamber and successfully assemble a s’more. As previously mentioned, the marshmallow detection was semi-functional, semi-flawed. The termination of the program often needed to be forced by user input (pressing a switch). The assembly functionality worked well consistently. It was easy to load, grabbed the marshmallow off the toasting rod, and returned the user the completed s’more outside the toasting assembly.
We learned many new skills and tricks from doing this project. There were novel mechanical challenges, such as how to deal with a 110V resistive heating element safely and cautiously. Luckily, there were never any damages or problems with the heating element due to our safe handling. We also learned how to use OpenCV, the package used to analyze our images and draw conclusions from them. This is a powerful tool that we look forward to further exploring in the future.
Safety played a big part in the development of this project. We spent a lot of time protecting the heating element from drawing too much power. We did this by using a relay and a manually switched power strip as a backup. Once it was safely drawing power, we insulated the chassis of the whole machine in fiberglass and aluminum foil, to prevent heat from escaping too much to the electronics. Finally, we had a manual override in the program to retrieve the marshmallow if the user thought it was getting too toasted. We ended up not having any heating accidents throughout production of SMORE. 
RESOURCES
References:
[1] (for raspberry pi) – https://www.raspberrypi.org/products/raspberry-pi-3-model-b-plus[2] (continuous rotation servo)
[3] for piCamera) – https://www.raspberrypi.org/products/camera-module-v2/p
[4] reference claw https://www.youmagine.com/designs/simple-robot-claw
[5] (stepper motor) – https://www.amazon.com/Stepper-Bipolar-Printer-Machine-Robotics/dp/B07BKRWK1Q/ref=sr_1_5?ie=UTF8&qid=1544357387&sr=8-5&keywords=stepper+motor+nema17
[6] (OpenCV) – https://opencv.org
[7] (stepper driver) – https://www.amazon.com/Qunqi-2Packs-Controller-Stepper-Arduino/dp/B01M29YK5U/ref=sr_1_2?s=hi&ie=UTF8&qid=1544356979&sr=1-2&keywords=Qunqi+L298N+Motor+Drive+Controller+Board+Module+Dual+H+Bridge+DC+Stepper+For+Arduino
Microchip Technology | www.microchip.com
Raspberry Pi Foundation | www.raspberrypi.org
PUBLISHED IN CIRCUIT CELLAR MAGAZINE • JUNE 2020 #359 – Get a PDF of the issue
Sponsor this Article
Katie Bradford ( keb278@cornell.edu) is a senior electrical and computer engineering major at Cornell University. She enjoys everything robotics, mechatronics and prototyping. While not in class, she can be found helping others
make projects in the school’s maker spaces, practicing for her dance team and cooking with friends.
Jared Gallina (jdg272@cornell.edu) is a senior majoring in electrical and computer engineering at Cornell. His academic interests are robotics and embedded systems. Outside of class he is an instructor for outdoor education classes at Cornell.
