Mallik Moturi

Edge devices with connectivity or Internet of Things (IoT) are having a significant impact on how businesses effectively use data to manage their products and services to grow and scale up. In this two part series, I will first discuss some of the challenges of the current approach, such as scalability of deployment due to escalating expenses from recurring network and cloud costs. Furthermore, the legacy processors and devices deployed in the last decade were not designed to meet this challenge. Part two of At the Edge will discuss how a new generation of strong and efficient edge AI processors and rapid evolution in new technologies are rising to solve these challenges.

An example of the unsustainability of the current approach to businesses is the recent announcement by Andy Jessy (CEO, Amazon) to reduce billions of dollars in cost to recover the estimated $5 billion in losses annually. A significant portion of Alexa voice expenses is in managing the hundred million echo devices from the cloud. Furthermore, most of these devices were sold at cost and deployed with expensive, sub-optimal processors with no in-built AI capabilities.

Data analytics on the edge device reduces cost and improves data security Increasingly, businesses are deploying sensors at the edge of the network. This is good news for those that need quick intelligence. But the question is, can this data be utilized in ways that are smart and cost-effective?

For sensors connected to the IoT devices, one of the more popular ways to analyze data stream has been to first send data directly to the cloud. This was due to the lack of sufficient AI processing capability at the edge. Cloud computation can be beneficial due to the abundance of high-performance servers available but is also costly. This is because sending data to a central server takes network bandwidth, recurring storage cost, server (CPU or GPU) cost, all the while adding processing delays and latency.

One way to reduce network bandwidth consumption is to compress the data before sending it to the cloud where decompression of the data can be done on cloud servers. However, compressing sensor data takes considerable energy at the edge and is often not suitable for many always-on, battery operated edge devices.

Sensor data analytics at the edge using advanced AI techniques is now possible. Edge devices are evolving rapidly, where the AI capability is doubling every year, and more complex models can be implemented while keeping within similar low power as in the prior generations. AI at the edge can also improve security, especially if the data is too sensitive to be allowed to be transmitted over a network. This can help businesses that are regulated by privacy laws or deal with sensitive data.

A recent use case of edge AI is for electric vehicles (EV). Performance of batteries for fleets of EVs are being monitored at the edge, significantly reducing costs to the operator. It also helps ensure that customer-sensitive vehicle data is not shared over a network or stored in the cloud.

Edge AI can also help reduce network energy costs. This can be especially important for businesses that are dealing with large numbers of IoT devices as in the aforementioned use case for EV.

In conclusion, edge AI is a promising way to reduce network costs, reduce cloud costs, and improve data security. The benefits of edge analytics include quicker decision making, improved security, and a better customer experience.

Embedded devices collect data from various sensors Embedded systems collect data from various sensors and other sources in the field. These systems then transmit real-world information from multiple sensors in real time to the cloud.

Such embedded systems fitted with edge AI capabilities can, for example, help companies improve their supply chain, spot component failures and improve the performance of production systems. These systems can monitor and collect data in real time to improve product quality and effectiveness. The benefits are not limited to manufacturing, as embedded systems are finding their way into many categories of automotive. They can also be used for safety, such as in anti-lock braking systems, detection of distracted driving, or for security purposes, such as theft or tamper detection,

While the IoT has already transformed industries like manufacturing, there is still much to be done. For instance, more IoT devices will need to be able to process data in real time, which requires them to be always-on with ultra-low power consumption. This is the case for always-on cameras, smart meters, and more. The challenge will be to be able to run concurrent sensor analytics accurately with high confidence, which requires high-performing edge AI at the device level.

Challenges in adapting deep learning (DL) algorithms for use on microcontroller (MCU) based edge devices Developing a machine learning model on intelligent edge devices has the potential to enable them to make better predictions and adapt to new data. However, training these models on low-power microcontrollers is challenging. They have limited memory capacity and limited compute power.

Adapting deep learning (DL) algorithms for use on MCU based embedded platforms requires careful hardware and software co-design. The size of the DL model must fit within the constraints of the MCU powered device. In addition, energy efficiency is a critical factor. The computational cost of the DL algorithm is also important.

Most commercial MCUs are powered by ARM Cortex-M cores and typically have 1 MB of flash memory and 256 KB of RAM. These devices are suited to running lightweight inference algorithms which require a small memory space and a short battery life. Newer cores support Single Instruction Multiple Data (SIMD) operations. The ARM CMSIS-NN library based on these operations offers optimized inference code for a variety of neural network models. However, the core architecture is tuned for scalar operations where instructions and data are fetched from memory prior to every operation.

One of the major challenges of deploying DL algorithms on MCU based embedded platforms is battery life. In addition, the communication speed of embedded devices is often limited. This also leads to latency and performance issues. This means that each DL model needs to be hand tuned and optimized to have low inference latency. To achieve this, AI model developers resort to using a model-reduction framework with methods such as: binarisation, model pruning, and parameter quantization.

To address the memory and computational bottlenecks of MCUs, the TinyML initiative enables light-weight neural networks to be deployed on IoT centric MCUs where developers have to reduce the overall size or memory cost of each network. A variety of frameworks are being used for this purpose. One framework uses C++ and TensorFlow Lite Micro (TFLM) to deploy a machine learning model. Another new framework called micro-TVM allows for the deployment of optimized models on MCU based edge devices. It can be used to reduce memory consumption by splitting the neural network layer into multiple series of tiles.

These challenges and complexities to address them have led to an evolution in the architectures for DL on edge devices where parallel processing of data streams, smaller bit width for weights and activations, in-memory or at-memory computations, and tiling have been applied.

In Part 2 of this article I will describe the expected growth in Deep Neural Network (DNN) processors on edge devices. I will also describe how these processors are evolving rapidly and growing stronger with every new generation to accommodate larger and more complex networks that are needed to handle the smart interface processing right on the edge which is not possible on today's MCUs. This strong edge AI capability will allow the cloud AI to handle what it is really good at, namely the large-scale application processing, generative AI, or natural language processing (NLP) which result in outcomes that users need.

About Syntiant Located in Irvine, California, Syntiant Corp. is focused on deep learning solutions for low-power, intelligent sensor processing in Edge AI devices. The second-generation Neural Decision Processors, the NDP120 & the NDP200, have been in volume production since summer 2021 and 2022 respectively.

Syntiant will be demonstrating many interesting edge AI applications at the upcoming Consumer Electronics Show in Las Vegas. Ask me about Syntiant AI processors & solutions, and how we can "Make it Syntiant" together.

Click on the banner above to visit Syntiant CES 2023 site to see our demonstrations or book an appointment with me.

Mallik Moturi

San Diego, California, United States

3K followers 500+ connections

View mutual connections with Mallik

Welcome back

Email or phone

Password

Forgot password?

or

New to LinkedIn? Join now

or

New to LinkedIn? Join now

About

Experience

Syntiant Corp.

-

-

-

-

-

-

-

-

-

Education

University of Pennsylvania

Recommendations received

Shelly Begun

Brad Suessmith

View Mallik’s full profile

Explore collaborative articles

Others named Mallik Moturi

Mallik Moturi

Add new skills with these courses

GitHub for Data Science Job Seekers

Tech on the Go: From Software Developer to Architect

Learning Verilog for FPGA Development

Mallik Moturi

San Diego, California, United States 3K followers 500+ connections

See your mutual connections View mutual connections with Mallik Sign in Welcome back Email or phone Password Show Forgot password? Sign in or New to LinkedIn? Join now or New to LinkedIn? Join now

About

Experience

Syntiant Corp.

-

-

-

-

-

-

-

-

-

Education

University of Pennsylvania

Recommendations received

Shelly Begun

Brad Suessmith

View Mallik’s full profile

Explore collaborative articles

Others named Mallik Moturi

Mallik Moturi

Add new skills with these courses

GitHub for Data Science Job Seekers

Tech on the Go: From Software Developer to Architect

Learning Verilog for FPGA Development

San Diego, California, United States

3K followers 500+ connections

View mutual connections with Mallik

Welcome back

Email or phone

Password

Forgot password?

or

New to LinkedIn? Join now

or

New to LinkedIn? Join now