Serial Memory Controller Meets AI and Machine Learning Needs

Microchip Technology has announced what is claims is the industry’s first commercially available serial memory controller. The SMC 1000 8x25G enables CPUs and other compute-centric SoCs to utilize four times the memory channels of parallel attached DDR4 DRAM within the same package footprint. Microchip’s serial memory controllers deliver higher memory bandwidth and media independence to these compute-intensive platforms with ultra-low latency.

As the computational demands of artificial intelligence (AI) and machine learning workloads accelerate, traditional parallel attached DRAM memory has presented a major roadblock for next-generation CPUs, which require an increased number of memory channels to deliver more memory bandwidth.

As the number of processing cores within CPUs has risen, the average memory bandwidth available to each processing core has decreased because CPU and SoC devices cannot scale the number of parallel DDR interfaces on a single chip to meet the needs of the increasing core count.

The SMC 1000 8x25G interfaces to the CPU via 8-bit Open Memory Interface (OMI)-compliant 25 Gbps lanes and bridges to memory via a 72-bit DDR4 3200 interface. The result is a significant reduction in the required number of host CPU or SoC pins per DDR4 memory channel, allowing for more memory channels and increasing the memory bandwidth available.

A CPU or SoC with OMI support can utilize a broad set of media types with different cost, power and performance metrics without having to integrate a unique memory controller for each type. In contrast, CPU and SoC memory interfaces today are typically locked to specific DDR interface protocols, such as DDR4, at specific interface rates. The SMC 1000 8x25G is the first memory infrastructure product in Microchip’s portfolio that enables the media-independent OMI interface.

Data center application workloads require OMI-based DDIMM memory products to deliver the same high-performance bandwidth and low latency results of today’s parallel-DDR based memory products. Microchip’s SMC 1000 8x25G features an innovative low latency design that delivers less than four ns incremental latency over a traditional integrated DDR controller with LRDIMM. This results in OMI-based DDIMM products having virtually identical bandwidth and latency performance to comparable LRDIMM products.

SMART Modular, Micron and Samsung Electronics are building multiple pin-efficient 84-pin Differential Dual-Inline Memory Modules (DDIMM) with capacities ranging from 16 GB to 256 GB, conforming to the draft JEDEC DDR5 standard DDIMM form factor. These DDIMMs will leverage the SMC 1000 8x25G and will seamlessly plug into any OMI-compliant 25 Gbps interface.

To support customers building systems that are compliant with the OMI standard, the SMC 1000 comes with design-in collateral and ChipLink diagnostic tools that provide extensive debug, diagnostics, configuration and analysts tools with an intuitive GUI.  The SMC 1000 8x25G is sampling now.

Microchip Technology | www.microchip.com

February Circuit Cellar: Sneak Preview

The February issue of Circuit Cellar magazine is coming soon. We’ve raised up a bumper crop of in-depth embedded electronics articles just for you, and packed ’em into our 84-page magazine.

Not a Circuit Cellar subscriber?  Don’t be left out! Sign up today:

 

Here’s a sneak preview of February 2019 Circuit Cellar:

MCUs ARE EVERYWHERE, DOING EVERYTHING

Electronics for Automotive Infotainment
As automotive dashboard displays get more sophisticated, information and entertainment are merging into so-called infotainment systems. That’s driving a need for powerful MCU- and MPU-based solutions that support the connectivity, computing and interfacing needs particular to these system designs. In this article, Circuit Cellar’s Editor-in-Chief, Jeff Child, looks at the technology and trends feuling automotive infotainment.

Inductive Sensing with PSoC MCUs
Inductive sensing is shaping up to be the next big thing for touch technology. It’s suited for applications involving metal-over-touch situations in automotive, industrial and other similar systems. In his article, Nishant Mittal explores the science and technology of inductive sensing. He then describes a complete system design, along with firmware, for an inductive sensing solution based on Cypress Semiconductor’s PSoC microcontroller.

Build a Self-Correcting LED Clock
In North America, most radio-controlled clocks use WWVB’s transmissions to set the correct time. WWVB is a Colorado-based time signal radio station near. Learn how Cornell graduates Eldar Slobodyan and Jason Ben Nathan designed and built a prototype of a Digital WWVB Clock. The project’s main components include a Microchip PIC32 MCU, an external oscillator and a display.

WE’VE GOT THE POWER

Product Focus: ADCs and DACs
Analog-to-digital converters (ADCs) and digital-to-analog converters (DACs) are two of the key IC components that enable digital systems to interact with the real world. Makers of analog ICs are constantly evolving their DAC and ADC chips pushing the barriers of resolution and speeds. This new Product Focus section updates readers on this technology and provides a product album of representative ADC and DAC products.

Building a Generator Control System
Three phase electrical power is a critical technology for heavy machinery. Learn how US Coast Guard Academy students Kent Altobelli and Caleb Stewart built a physical generator set model capable of producing three phase electricity. The article steps through the power sensors, master controller and DC-DC conversion design choices they faced with this project.

EMBEDDED COMPUTING FOR YOUR SYSTEM DESIGN

Non-Standard Single Board Computers
Although standard-form factor embedded computers provide a lot of value, many applications demand that form take priority over function. That’s where non-standard boards shine. The majority of non-standard boards tend to be extremely compact, and well suited for size-constrained system designs. Circuit Cellar Chief Editor Jeff Child explores the latest technology trends and product developments in non-standard SBCs.

Thermal Management in machine learning
Artificial intelligence and machine learning continue to move toward center stage. But the powerful processing they require is tied to high power dissipation that results in a lot of heat to manage. In his article, Tom Gregory from 6SigmaET explores the alternatives available today with a special look at cooling Google’s Tensor Processor Unit 3.0 (TPUv3) which was designed with machine learning in mind.

… AND MORE FROM OUR EXPERT COLUMNISTS

Bluetooth Mesh (Part 1)
Wireless mesh networks are being widely deployed in a wide variety of settings. In this article, Bob Japenga begins his series on Bluetooth mesh. He starts with defining what a mesh network is, then looks at two alternatives available to you as embedded systems designers.

Implementing Time Technology
Many embedded systems need to make use of synchronized time information. In this article, Jeff Bachiochi explores the history of time measurement and how it’s led to NTP and other modern technologies for coordinating universal date and time. Using Arduino and the Espressif System’s ESP32, Jeff then goes through the steps needed to enable your embedded system to request, retrieve and display the synchronized date and time to a display.

Infrared Sensors
Infrared sensing technology has broad application ranging from motion detection in security systems to proximity switches in consumer devices. In this article, George Novacek looks at the science, technology and circuitry of infrared sensors. He also discusses the various types of infrared sensing technologies and how to use them.

The Art of Voltage Probing
Using the right tool for the right job is a basic tenant of electronics engineering. In this article, Robert Lacoste explores one of the most common tools on an engineer’s bench: oscilloscope probes, and in particular the voltage measurement probe. He looks and the different types of voltage probes as well as the techniques to use them effectively and safely.

Tool Extension Enables Neural Networking on STM32 MCUs

STMicroelectronics has extended its STM32CubeMX ecosystem by adding advanced Artificial Intelligence (AI) features.  AI uses trained artificial neural networks to classify data signals from motion and vibration sensors, environmental sensors, microphones and image sensors, more quickly and efficiently than conventional handcrafted signal processing. With STM32Cube.AI, developers can now convert pre-trained neural networks into C-code that calls functions in optimized libraries that can run on STM32 MCUs.
STM32Cube.AI comes together with ready-to-use software function packs that include example code for human activity recognition and audio scene classification. These code examples are immediately usable with the ST SensorTile reference board and the ST BLE Sensor mobile app. Additional support such as engineering services is available for developers through qualified partners inside the ST Partner Program and the dedicated AI and Machine Learning (ML) STM32 online community. ST will demonstrate applications developed using STM32Cube.AI running on STM32 MCUs this week in a private suite at CES, the Consumer Electronics Show, in Las Vegas, January 8-12.

The STM32Cube.AI extension pack can be downloaded inside ST’s STM32CubeMX MCU configuration and software code-generation ecosystem. Today, the tool supports Caffe, Keras (with TensorFlow backend), Lasagne, ConvnetJS frameworks and IDEs including those from Keil, IAR and System Workbench.

The FP-AI-SENSING1 software function pack provides examples of code to support end-to-end motion (human-activity recognition) and audio (audio-scene classification) applications based on neural networks. This function pack leverages ST’s SensorTile reference board to capture and label the sensor data before the training process. The board can then run inferences of the optimized neural network. The ST BLE Sensor mobile app acts as the SensorTile’s remote control and display.

The comprehensive toolbox consisting of the STM32Cube.AI mapping tool, application software examples running on small-form-factor, battery-powered SensorTile hardware, together with the partner program and dedicated community support offers a fast and easy path to neural-network implementation on STM32 devices.

STMicroelectronics | www.st.com

 

January Circuit Cellar: Sneak Preview

Happy New Years! The January issue of Circuit Cellar magazine is coming soon. Don’t miss this 1st issue of Circuit Cellar 2019. Enjoy pages and pages of great, in-depth embedded electronics articles.

Not a Circuit Cellar subscriber?  Don’t be left out! Sign up today:

 

Here’s a sneak preview of January 2019 Circuit Cellar:

TRENDS & CHOICES IN EMBEDDED COMPUTING

Comms and Control for Drones
Consumer and commercial drones represent one of the most dynamic areas of embedded design today. Chip, board and system suppliers are offering improved ways for drones to do more processing on board the drone, while also providing solutions for implementing the control and communication subsystems in drones. This article by Circuit Cellar’s Editor-in-Chief Jeff Child looks at the technology and products available today that are advancing the capabilities of today’s drones.

Choosing an MPU/MCU for Industrial Design
By Microchip Technology’s Jacko Wilbrink
As MCU performance and functionality improve, the traditional boundaries between MCUs and microprocessor units (MPUs) have become less clear. In this article, Jacko examines the changing landscape in MPU vs. MCU capabilities, OS implications and the specifics of new SiP and SOM approaches for simplifying higher-performance computing requirements in industrial applications.

Product Focus: COM Express Boards
The COM Express architecture has found a solid and growing foothold in embedded systems. COM Express boards provide a complete computing core that can be upgraded when needed, leaving the application-specific I/O on the baseboard. This Product Focus section updates readers on this technology and provides a product album of representative COM Express products.

MICROCONTROLLERS ARE DOING EVERYTHING

Connecting USB to Simple MCUs
By Stuart Ball
Sometimes you want to connect a USB device such as a flash drive to a simple microcontroller. Problem is most MCUs cannot function as a USB host. In this article, Stuart steps through the technology and device choices that solve this challenge. He also puts the idea into action via a project that provides this functionality.

Vision System Enables Overlaid Images
By Daniel Edens and Elise Weir
In this project article, learn how these two Cornell students designed a system to overlay images from a visible light camera and an infrared camera. They use software running on a PIC32 MCU to interface the two types of cameras. The MCU does the computation to create the overlaid images, and displays them on an LCD screen.

DATA ACQUISITION AND MEASUREMENT

Data Acquisition Alternatives
By Jeff Child
While the fundamentals of data acquisition remain the same, its interfacing technology keeps evolving and changing. USB and PCI Express brought data acquisition off the rack, and onto the lab bench top. Today solutions are emerging that leverage Mini PCIe, Thunderbolt and remote web interfacing. Circuit Cellar’s Editor-in-Chief, Jeff Child, dives into the latest technology trends and product developments in data acquisition.

High-Side Current Sensing
By Jeff Bachiochi
Jeff says he likes being able to measure things—for example, being able to measure load current so he can predict how long a battery will last. With that in mind, he recently found a high-side current sensing device, Microchip’s EMC1701. In his article, Jeff takes you through the details of the device and how to make use of it in a battery-based system.

Power Analysis Capture with an MCU
By Colin O’Flynn
Low-cost microcontrollers integrate many powerful peripherals in them. You can even perform data capture directly to internal memory. In his article, Colin uses the ChipWhisperer-Nano as a case study in how you might use such features which would otherwise require external programmable logic.

TOOLS AND TECHNIQUES FOR EMBEDDED SYSTEM DESIGN

Easing into the IoT Cloud (Part 2)
By Brian Millier
In Part 1 of this article series Brian examined some of the technologies and services available today enabling you to ease into the IoT cloud. Now, in Part 2, he discusses the hardware features of the Particle IoT modules, as well as the circuitry and program code for the project. He also explores the integration of a Raspberry Pi solution with the Particle cloud infrastructure.

Hierarchical Menus for Touchscreens
By Aubrey Kagan
In his December article, Aubrey discussed his efforts to build a display subsystem and GUI for embedded use based on a Noritake touchscreen display. This time he shares how he created a menu system within the constraints of the Noritake graphical display system. He explains how he made good use of Microsoft Excel worksheets as a tool for developing the menu system.

Real Schematics (Part 2)
By George Novacek
The first part of this article series on the world of real schematics ended last month with wiring. At high frequencies PCBs suffer from the same parasitic effects as any other type of wiring. You can describe a transmission line as consisting of an infinite number of infinitesimal resistors, inductors and capacitors spread along its entire length. In this article George looks at real schematics from a transmission line perspective.

December Circuit Cellar: Sneak Preview

The December issue of Circuit Cellar magazine is coming soon. Don’t miss this last issue of Circuit Cellar in 2018. Pages and pages of great, in-depth embedded electronics articles prepared for you to enjoy.

Not a Circuit Cellar subscriber?  Don’t be left out! Sign up today:

 

Here’s a sneak preview of December 2018 Circuit Cellar:

AI, FPGAs and EMBEDDED SUPERCOMPUTING

Embedded Supercomputing
Gone are the days when supercomputing levels of processing required a huge, rack-based systems in an air-conditioned room. Today, embedded processors, FPGAs and GPUs are able to do AI and machine learning kinds of operation, enable new types of local decision making in embedded systems. In this article, Circuit Cellar’s Editor-in-Chief, Jeff Child, looks at these technology and trends driving embedded supercomputing.

Convolutional Neural Networks in FPGAs
Deep learning using convolutional neural networks (CNNs) can offer a robust solution across a wide range of applications and market segments. In this article written for Microsemi, Ted Marena illustrates that, while GPUs can be used to implement CNNs, a better approach, especially in edge applications, is to use FPGAs that are aligned with the application’s specific accuracy and performance requirements as well as the available size, cost and power budget.

NOT-TO-BE-OVERLOOKED ENGINEERING ISSUES AND CHOICES

DC-DC Converters
DC-DC conversion products must juggle a lot of masters to push the limits in power density, voltage range and advanced filtering. Issues like the need to accommodate multi-voltage electronics, operate at wide temperature ranges and serve distributed system requirements all add up to some daunting design challenges. This Product Focus section updates readers on these technology trends and provides a product gallery of representative DC-DC converters.

Real Schematics (Part 1)
Our magazine readers know that each issue of Circuit Cellar has several circuit schematics replete with lots of resistors, capacitors, inductors and wiring. But those passive components don’t behave as expected under all circumstances. In this article, George Novacek takes a deep look at the way these components behave with respect to their operating frequency.

Do you speak JTAG?
While most engineers have heard of JTAG or have even used JTAG, there’s some interesting background and capabilities that are so well know. Robert Lacoste examines the history of JTAG and looks at clever ways to use it, for example, using a cheap JTAG probe to toggle pins on your design, or to read the status of a given I/O without writing a single line of code.

PUTTING THE INTERNET-OF-THINGS TO WORK

Industrial IoT Systems
The Industrial Internet-of-Things (IIoT) is a segment of IoT technology where more severe conditions change the game. Rugged gateways and IIoT edge modules comprise these systems where the extreme temperatures and high vibrations of the factory floor make for a demanding environment. Here, Circuit Cellar’s Editor-in-Chief, Jeff Child, looks at key technology and product drives in the IIoT space.

Internet of Things Security (Part 6)
Continuing on with his article series on IoT security, this time Bob Japenga returns to his efforts to craft a checklist to help us create more secure IoT devices. This time he looks at developing a checklist to evaluate the threats to an IoT device.

Applying WebRTC to the IoT
Web Real-time Communications (WebRTC) is an open-source project created by Google that facilitates peer-to-peer communication directly in the web browser and through mobile applications using application programming interfaces. In her article, Callstats.io’s Allie Mellen shows how IoT device communication can be made easy by using WebRTC. With WebRTC, developers can easily enable devices to communicate securely and reliably through video, audio or data transfer.

WI-FI AND BLUETOOTH IN ACTION

IoT Door Security System Uses Wi-Fi
Learn how three Cornell students, Norman Chen, Ram Vellanki and Giacomo Di Liberto, built an Internet connected door security system that grants the user wireless monitoring and control over the system through a web and mobile application. The article discusses the interfacing of a Microchip PIC32 MCU with the Internet and the application of IoT to a door security system.

Self-Navigating Robots Use BLE
Navigating indoors is a difficult but interesting problem. Learn how these two Cornell students, Jane Du and Jacob Glueck, used Received Signal Strength Indicator (RSSI) of Bluetooth Low Energy (BLE) 4.0 chips to enable wheeled, mobile robots to navigate towards a stationary base station. The robot detects its proximity to the station based on the strength of the signal and moves towards what it believes to be the signal source.

IN-DEPTH PROJECT ARTICLES WITH ALL THE DETAILS

Sun Tracking Project
Most solar panel arrays are either fixed-position, or have a limited field of movement. In this project article, Jeff Bachiochi set out to tackle the challenge of a sun tracking system that can move your solar array to wherever the sun is coming from. Jeff’s project is a closed-loop system using severs, opto encoders and the Microchip PIC18 microcontroller.

Designing a Display System for Embedded Use
In this project article, Aubrey Kagan takes us through the process of developing an embedded system user interface subsystem—including everything from display selection to GUI development to MCU control. For the project he chose a 7” Noritake GT800 LCD color display and a Cypress Semiconductor PSoC5LP MCU.

FPGA Solutions Evolve to Meet AI Needs

Brainy System ICs

Long gone now are the days when FPGAs were thought of as simple programmable circuitry for interfacing and glue logic. Today, FPGAs are powerful system chips with on-chip processors, DSP functionality and high-speed connectivity.

By Jeff Child, Editor-in-Chief

Today’s FPGAs have now evolved to the point that calling them “systems-on-chips” is redundant. It’s now simply a given that the high-end lines of the major FPGA vendors have general-purpose CPU cores on them. Moreover, the flavors of signal processing functionality on today’s FPGA chips are ideally suited to the kind of system-oriented DSP functions used in high-end computing. And even better, they’ve enabled AI (Artificial Intelligence) and Machine Learning kinds of functionalities to be implemented into much smaller, embedded systems.

In fact, over the past 12 months, most of the leading FPGA vendors have been rolling out solutions specifically aimed at using FPGA technology to enable AI and machine learning in embedded systems. The two main FPGA market leaders Xilinx and Intel’s Programmable Solutions Group (formerly Altera) have certainly embraced this trend, as have many of their smaller competitors like Lattice Semiconductor and QuickLogic. Meanwhile, specialists in so-called e-FPGA technology like Archonix and Flex Logix have their own compelling twist on FPGA system computing.

Project Brainwave

Exemplifying the trend toward FPGAs facilitating AI processing, Intel’s high-performance line of FPGAs is its Stratix 10 family. According to Intel, the Stratix 10 FPGAs are capable of 10 TFLOPS, or 10 trillion floating point operations per second (Figure 1). In May Microsoft announced its Microsoft debuted its Azure Machine Learning Hardware Accelerated Models powered by Project Brainwave integrated with the Microsoft Azure Machine Learning SDK. Azure’s architecture is developed with Intel FPGAs and Intel Xeon processors.

Figure 1
Stratix 10 FPGAs are capable of 10 TFLOPS or 10 trillion floating point operations per second.

Intel says its FPGA-powered AI is able to achieve extremely high throughput that can run ResNet-50, an industry-standard deep neural network requiring almost 8 billion calculations without batching. This is possible using FPGAs because the programmable hardware—including logic, DSP and embedded memory—enable any desired logic function to be easily programmed and optimized for area, performance or power. And because this fabric is implemented in hardware, it can be customized and can perform parallel processing. This makes it possible to achieve orders of magnitudes of performance improvements over traditional software or GPU design methodologies.

In one application example, Intel cites an effort where Canada’s National Research Council (NRC) is helping to build the next-generation Square Kilometer Array (SKA) radio telescope to be deployed in remote regions of South Africa and Australia, where viewing conditions are most ideal for astronomical research. The SKA radio telescope will be the world’s largest radio telescope that is 10,000 times faster with image resolution 50 times greater than the best radio telescopes we have today. This increased resolution and speed results in an enormous amount of image data that is generated by these telescopes, processing the equivalent of a year’s data on the Internet every few months.

NRC’s design embeds Intel Stratix 10 SX FPGAs at the Central Processing Facility located at the SKA telescope site in South Africa to perform real-time processing and analysis of collected data at the edge. High-speed analog transceivers allow signal data to be ingested in real time into the core FPGA fabric. After that, the programmable logic can be parallelized to execute any custom algorithm optimized for power efficiency, performance or both, making FPGAs the ideal choice for processing massive amounts of real-time data at the edge.

ACAP for Next Gen

For its part, Xilinx’s high-performance product line is its Virtex UltraScale+ device family (Figure 2). According to the company, these provide the highest performance and integration capabilities in a FinFET node, including the highest signal processing bandwidth at 21.2 TeraMACs of DSP compute performance. They deliver on-chip memory density with up to 500 Mb of total on-chip integrated memory, plus up to 8 GB of HBM Gen2 integrated in-package for 460 GB/s of memory bandwidth. Virtex UltraScale+ devices provide capabilities with integrated IP for PCI Express, Interlaken, 100G Ethernet with FEC and Cache Coherent Interconnect for Accelerators (CCIX).

Figure 2
Virtex UltraScale+ FPGAs provide a signal processing bandwidth at 21.2 TeraMACs. They deliver on-chip memory density with up to 500 Mb of total on-chip integrated memory, plus up to 8 GB of HBM Gen2 integrated in-package for 460 GB/s of memory bandwidth.

Looking to the next phase of system performance, Xilinx in March announced its strategy toward a new FPGA product category it calls its adaptive compute acceleration platform (ACAP). Touted as going beyond the capabilities of an FPGA, an ACAP is a highly integrated multi-core heterogeneous compute platform that can be changed at the hardware level to adapt to the needs of a wide range of applications and workloads. An ACAP’s adaptability, which can be done dynamically during operation, delivers levels of performance and performance per-watt that is unmatched by CPUs or GPUs, says Xilinx… …

Read the full article in the August 337 issue of Circuit Cellar

Don’t miss out on upcoming issues of Circuit Cellar. Subscribe today!

Note: We’ve made the October 2017 issue of Circuit Cellar available as a free sample issue. In it, you’ll find a rich variety of the kinds of articles and information that exemplify a typical issue of the current magazine.

Dev Kit Enables Cars to Express Their Emotions

Renesas Electronics has announced that it has developed a development kit for its R-Car that takes advantage of “emotion engine”, an artificial sensibility and intelligence technology pioneered by cocoro SB Corp. The new development kit enables cars with the sensibility to read the driver’s emotions and optimally respond to the driver’s needs based on their emotional state.

The development kit includes cocoro SB’s emotion engine, which was developed leveraging its sensibility technology to recognize emotional states such as confidence or uncertainty based on the speech of the driver. The car’s response to the driver’s emotional state is displayed by a new driver-attentive user interface (UI) implemented in the Renesas R-Car system-on-chip (SoC). Since it is possible for the car to understand the driver’s words and emotional state, it can provide the appropriate response that ensures optimal driver safety.

20170719-verbal-emotion-recognition-engine-st

As this technology is linked to artificial intelligence (AI) based machine learning, it is possible for the car to learn from conversations with the driver, enabling it to transform into a car that is capable of providing the best response to the driver. Renesas plans to release the development kit later this year.

Renesas  demonstrated its connected car simulator incorporating the new development kit based on cocoro SB’s emotion engine at the SoftBank World 2017 event earlier this month in held by SoftBank at the Prince Park Tower Tokyo.

Renesas considers the driver’s emotional state, facial expression and eyesight direction as key information that combines with the driver’s vital signs to improve the car and driver interface, placing drivers closer to the era of self-driving cars. For example, if the car can recognize the driver is experiencing an uneasy emotional state, even if he or she has verbally accepted the switch to hands free autonomous-driving mode, it is possible for the car to ask the driver “would you prefer to continue driving and not switch to autonomous-driving mode for now?” Furthermore, understanding the driver’s emotions enables the car to control vehicle speed according to how the driver is feeling while driving at night in autonomous-driving mode. By providing carmakers and IT companies with the development kit that takes advantage of this emotion engine, Renesas hopes to expand the possibilities for this service model to the development of new interfaces between cars and drivers and other mobility markets that can take advantage of emotional state information. Based on the newly-launched Renesas autonomy, a new advanced driving assistance systems (ADAS) and automated driving platform, Renesas enables a safe, secure, and convenient driving experience by providing next-generation solutions for connected cars.

Renesas Electronics America | www.renesas.com

Natural Human-Computer Interaction

Recent innovations in both hardware and software have brought on a new wave of interaction techniques that depart from mice and keyboards. The widespread adoption of smartphones and tablets with capacitive touchscreens shows people’s preference to directly manipulate virtual objects with their hands.

Going beyond touch-only interaction, the Microsoft Kinect sensor enables users to play

This shows the hand tracking result from Kinect data. The red regions are our tracking results and the green lines are the skeleton tracking results from the Kinect SDK (based on data from the ChAirGest corpus: https://project.eia-fr.ch/chairgest/Pages/Overview.aspx).

This shows the hand tracking result from Kinect data. The red regions are our tracking results and the green lines are the skeleton tracking results from the Kinect SDK (based on data from the ChAirGest corpus: https://project.eia-fr.ch/chairgest/Pages/Overview.aspx).

games with their entire body. More recently, Leap Motion’s new compact sensor, consisting of two cameras and three infrared LEDs, has opened up the possibility of accurate fingertip tracking. With Project Glass, Google is pioneering new technology in the wearable human-computer interface. Other new additions to wearable technology include Samsung’s Galaxy Gear Smartwatch and Apple’s rumored iWatch.

A natural interface reduces the learning curve, or the amount of time and energy a person requires to complete a particular task. Instead of a user learning to communicate with a machine through a programming language, the machine is now learning to understand the user.

Hardware advancements have led to our clunky computer boxes becoming miniaturized, stylish sci-fi-like phones and watches. Along with these shrinking computers come ever-smaller sensors that enable a once keyboard-constrained computer to listen, see, and feel. These developments pave the way to natural human-computer interfaces.
If sensors are like eyes and ears, software would be analogous to our brains.

Understanding human speech and gestures in real time is a challenging task for natural human-computer interaction. At a higher level, both speech and gesture recognition require similar processing pipelines that include data streaming from sensors, feature extraction, and pattern recognition of a time series of feature vectors. One of the main differences between the two is feature representation because speech involves audio data while gestures involve video data.

For gesture recognition, the first main step is locating the user’s hand. Popular libraries for doing this include Microsoft’s Kinect SDK or PrimeSense’s NITE library. However, these libraries only give the coordinates of the hands as points, so the actual hand shapes cannot be evaluated.

Fingertip tracking using a Kinect sensor. The green dots are the tracked fingertips.

Our team at the Massachusetts Institute of Technology (MIT) Computer Science and Artificial Intelligence Laboratory has developed methods that use a combination of skin-color and motion detection to compute a probability map of gesture salience location. The gesture salience computation takes into consideration the amount of movement and the closeness of movement to the observer (i.e., the sensor).

We can use the probability map to find the most likely area of the gesturing hands. For each time frame, after extracting the depth data for the entire hand, we compute a histogram of oriented gradients to represent the hand shape as a more compact feature descriptor. The final feature vector for a time frame includes 3-D position, velocity, and hand acceleration as well as the hand shape descriptor. We also apply principal component analysis to reduce the feature vector’s final dimension.

A 3-D model of pointing gestures using a Kinect sensor. The top left video shows background subtraction, arm segmentation, and fingertip tracking. The top right video shows the raw depth-mapped data. The bottom left video shows the 3D model with the white plane as the tabletop, the green line as the arm, and the small red dot as the fingertip.

The next step in the gesture-recognition pipeline is to classify the feature vector sequence into different gestures. Many machine-learning methods have been used to solve this problem. A popular one is called the hidden Markov model (HMM), which is commonly used to model sequence data. It was earlier used in speech recognition with great success.

There are two steps in gesture classification. First, we need to obtain training data to learn the models for different gestures. Then, during recognition, we find the most likely model that can produce the given observed feature vectors. New developments in the area involve some variations in the HMM, such as using hierarchical HMM for real-time inference or using discriminative training to increase the recognition accuracy.

Ying Yin

Ying Yin is a PhD candidate and a Research Assistant at the Massachusetts Institute of Technology (MIT) Computer Science and Artificial Intelligence Laboratory. Originally from Suzhou, China, Ying received her BASc in Computer Engineering from the University of British Columbia in Vancouver, Canada, in 2008 and an MS in Computer Science from MIT in 2010. Her research focuses on applying machine learning and computer vision methods to multimodal human-computer interaction. Ying is also interested in web and mobile application development. She has won awards in web and mobile programming competitions at MIT.

Currently, the newest development in speech recognition at the industry scale is a method called deep learning. Earlier machine-learning methods require careful selection of feature vectors. The goal of deep learning is automatic discovery of powerful features from raw input data. So far, it has shown promising results in speech recognition. It can possibly be applied to gesture recognition to see whether it can further improve accuracy.

As component form factors shrink, sensor resolutions grow, and recognition algorithms become more accurate, natural human-computer interaction will become more and more ubiquitous in our everyday life.