The Rise of Scalable AI SoCs for the IoT Device Edge

The landscape of computing is undergoing a profound transformation, with Artificial Intelligence (AI) at its forefront. This shift is particularly evident at the device edge, where traditional System-on-Chip (SoCs) implementations are being reimagined to effectively support demanding AI and machine learning (ML) workloads.

This evolution necessitates the development of a new class of AI-capable Microprocessor Units (MPUs), and Microcontroller Units (MCUs), which are becoming indispensable for processing functions at the device's edge.

The Evolution of Compute and the Shift to the Edge

Historically, the compute environment relating to embedded systems was characterized by significant fragmentation, marked by a proliferation of various custom operating systems and firmware.

However, over the past 15 years or so, particularly within the Internet of Things (IoT) domain, efforts have resulted in the adoption of Linux and FreeRTOS as preferred base operating systems, with firmware layers streamlining through standards like UEFI (Unified Extensible Firmware Interface), implementations such as U-Boot, and supported via open-source compiler toolchains, such as GCC and CLANG/LLVM.

Despite this software standardization, silicon architectures have not achieved the same level of uniformity, remaining highly diverse depending on the specific IoT segment, such as consumer, industrial, automotive, or healthcare, among others. Edge computing originated in the late 1990s with Content Delivery Networks (CDNs), which cached static web content at geographically distributed edge servers to reduce latency. This concept evolved with the development of Multi-access Edge Computing (MEC), which aimed to extend compute and storage capabilities closer to end-users at cellular base stations, to enable faster processing for mobile applications like video streaming, IoT, and autonomous systems.

Cloud providers played a crucial role in bringing edge compute and hybrid computing to fruition, establishing the necessary ecosystems. Today, the next wave of disruption in compute is unequivocally fueled by the integration of AI.

While AI training primarily occurs in the cloud, typically utilizing powerful GPUs, the focus is increasingly shifting to inferencing, which primarily involves working with pre-trained models. This inferencing capability is progressively creeping down into different levels of edge compute, moving closer to the IoT device edge.

The Challenges of AI Workloads at the Device Edge

Deploying AI at the IoT edge creates multiple challenges. AI workloads and their associated software encounter highly heterogeneous and diverse, embedded system architectures. This leads to considerable marketplace confusion, a lack of clear blueprints, and an absence of best practices for implementing AI functions effectively.

Smaller, cut-down versions of traditional AI hardware, such as GPUs, are often too power-hungry and expensive when used for inferencing workloads on resource-constrained IoT systems. This means there are no clear accelerator choices for handling inferencing functions at the edge. This also means the industry is missing design standards and best practices for enabling a consistent user experience for IoT AI.

Also, many silicon players have primarily focused on larger, more homogeneous markets, such as automotive, industrial and smartphones. They tend to repurpose existing silicon for the IoT, instead of designing purpose-built solutions that deliver the right balance of performance, power and cost. Compounding this hardware fragmentation, various edge silicon architectures are often powered by equally custom and proprietary AI software frameworks. This deepens the lack of standardization in the overall AI subsystem experience.

Reimagining SoCs: The New Class of Edge AI Silicon

To address these challenges, the market requires a fundamental rethinking of how to build scalable, performant AI-capable silicon for the IoT device edge. Synaptics is spearheading this new class of edge AI silicon with product lines designed to solve endemic fragmentation in IoT devices, and deliver additional intelligence at every power level. Key to this approach is the innovative hardware and software architecture that powers the Neural Processor Unit (NPU) subsystems within the SoC product lines, designed for scalability and Edge AI future-proofing:

• Transformer acceleration: Transformers are the foundational technology for large language models (LLMs) and are critical for multimodal AI interactions, such as natural language processing. Along with traditional Convolutional Neural Network (CNN)-based image, video and audio models, hardware support for accelerating transformers is essential for state-of-the-art models.
• Tightly coupled general purpose compute: Localized processing based on standard core compute can be used to efficiently run newer or unsupported operators, without offloading to host CPUs. This low-latency, high-performance approach allows customers to leverage their silicon investments for longer periods, safeguarding against rapidly shifting Edge AI processing paradigms.
• Tiered Memory: A combination of configurable internal memory and external system memory can be utilized to store compiled models, instructions and descriptors, input/output data and any intermediate data states, as needed.
• Offline Compilation: An advantageous approach where the compiler analyzes the model to be accelerated and "shapes" it ahead of runtime, for efficient execution.

The new SoC product lines, and the broader Synaptics Astra series of AI-native processors, are designed to deliver significant competitive advantages in overall performance-per-watt for Edge AI IoT workloads.

Addressing the Edge AI Software Fragmentation and Ecosystem Challenges

Despite progress in standardizing general embedded software (such as Linux, FreeRTOS, or Zephyr), the software stack for AI functions remains largely non-standardized and separate from the diverse silicon architectures. This is a significant hurdle, but one that companies, such as cloud service providers looking to establish a broad footprint for multimodal edge processing, can tackle since they have the scale to establish and drive standards and ecosystems.

In the AI Software Tech stack, some of the elements - e.g. model formats and runtimes such as TFLite (now called LiteRT), PyTorch and ONNX - have become more widely accepted as practice. But when it comes to Edge AI compilers, today it's a collection of custom, proprietary implementations closely tied to SoCs, which proliferate vendor lock-in.

The solution could lie with industry-defining efforts spearheaded by hyperscalers like Google, Meta, AWS, Alibaba and Baidu, in close partnership with silicon vendors, to enable platforms built on open-source tooling. Such broad partnerships are crucial to create large, open and developer-friendly ecosystems with material scale.

The recently announced collaboration between Synaptics and Google Research addresses fragmentation challenges across NPU subsystem design and tooling and aims to set a new standard for AI application developer experiences at the IoT device edge. In addition to integrating scalable high-performance NPU IP, the effort centers on the IREE toolchain - an open-source, end-to-end compilation and runtime implementation based on the MLIR core framework. IREE includes support for inference acceleration on diverse hardware such as NPUs, CPUs and GPUs, and supports standard ML frameworks such as TFLite, PyTorch and ONNX.

A Holistic Approach to Edge AI

Ultimately, solving the complex challenges of AI at the IoT edge requires a holistic strategy that combines an open-source software approach with an innovative silicon architecture. This integrated strategy is designed to make AI less daunting and more accessible, empowering developers, users and customers within the vast IoT space to unlock the full potential of artificial intelligence directly at the device edge.