Microsoft Real-time AI Project Leverages FPGAs

At Hot Chips 2017 Microsoft unveiled a new deep learning acceleration platform, codenamed Project Brainwave. The system performs real-time AI. Real-time here means the system processes requests as fast as it receives them, with ultra-low latency. Real-time AI is becoming increasingly important as cloud infrastructures process live data streams, whether they be search queries, videos, sensor streams, or interactions with users.



The Project Brainwave system is built with three main layers: a high-performance, distributed system architecture; a hardware DNN engine synthesized onto FPGAs; and a compiler and runtime for low-friction deployment of trained models. Project Brainwave leverages the massive FPGA infrastructure that Microsoft has been deploying over the past few years. By attaching high-performance FPGAs directly to Microsoft’s datacenter network, they can serve DNNs as hardware microservices, where a DNN can be mapped to a pool of remote FPGAs and called by a server with no software in the loop. This system architecture both reduces latency, since the CPU does not need to process incoming requests, and allows very high throughput, with the FPGA processing requests as fast as the network can stream them.

Project Brainwave uses a powerful “soft” DNN processing unit (or DPU), synthesized onto commercially available FPGAs.  A number of companies—both large companies and a slew of startups—are building hardened DPUs.  Although some of these chips have high peak performance, they must choose their operators and data types at design time, which limits their flexibility.  Project Brainwave takes a different approach, providing a design that scales across a range of data types, with the desired data type being a synthesis-time decision. The design combines both the ASIC digital signal processing blocks on the FPGAs and the synthesizable logic to provide a greater and more optimized number of functional units.  This approach exploits the FPGA’s flexibility in two ways.  First, the developers have defined highly customized, narrow-precision data types that increase performance without real losses in model accuracy.  Second, they can incorporate research innovations into the hardware platform quickly (typically a few weeks), which is essential in this fast-moving space.  As a result, the Microsoft team achieved performance comparable to – or greater than – many of these hard-coded DPU chips but are delivering the promised performance today. At Hot Chips, Project Brainwave was demonstrated using Intel’s new 14 nm Stratix 10 FPGA.

Project Brainwave incorporates a software stack designed to support the wide range of popular deep learning frameworks. They support Microsoft Cognitive Toolkit and Google’s Tensorflow, and plan to support many others. They have defined a graph-based intermediate representation, to which they convert models trained in the popular frameworks, and then compile down to their high-performance infrastructure.

Microsoft |

Don’t Miss Our Bonus Newsletter: FPGA Technologies

As you know, Circuit Cellar’s newsletter covers four key themes each month. But August is a special month with a 5th Tuesday! As result, tomorrow coming to your inbox with be a special bonus newsletter theme: FPGA Technologies. In tomorrow’s newsletter you’ll get news about the products and technologies trends in the FPGA market. FPGAs have sv_gs_diagramevolved to become complete system chips. Today’s FPGAs pack in levels of processing, I/O and memory on one chip that once required several ICs or boards.

Also: We’ve added Drawings for Free Stuff to our weekly newsletters. Make sure you’ve subscribed to the newsletter so you can participate.

Already a Circuit Cellar Newsletter subscriber? Great!
You’ll get your “FPGA Technology” themed newsletter issue tomorrow.

Not a Circuit Cellar Newsletter subscriber?
Don’t be left out! Sign up now:

Remember, our new enhanced weekly CC Newsletter will switch its theme each week, so look for these in upcoming weeks:

Analog & Power. This newsletter content zeros in on the latest developments in analog and power technologies including DC-DC converters, AD-DC converters, power supplies, op-amps, batteries, and more.

Microcontroller Watch. This newsletter keeps you up-to-date on latest microcontroller news. In this section, we examine the microcontrollers along with their associated tools and support products.

IoT Technology Focus. The Internet-of-Things (IoT) phenomenon is rich with opportunity. This newsletter tackles news and trends about the products and technologies needed to build IoT implementations and devices.

Embedded Boards. Embedded boards are critical building blocks around which system developers can build all manor of intelligent systems. The focus here is on both standard and non-standard embedded computer boards.