Key points:

• To match the appropriate AI model for embedded applications, the first step is to clarify the purpose, including the functions of the target device and the product characteristics.
• Designers usually combine multiple compression techniques rather than using just one method. Common strategies include quantization, distillation, and pruning.
• Small language models ( SLM ) and compact edge models require tight system-level integration to achieve optimal efficiency.

Today, the AI industry faces two distinctly different challenges. On one hand, there are massive AI data centers and AI factories that are seeking more efficient ways to handle large language models with high parameters (LLM), where bandwidth and memory are key bottlenecks. On the other hand, there are small AI models running on edge devices, which do not have sufficient space, power consumption, or memory to support more complex models with more parameters (LLM).

Imagination Technologies CTO Ed Plowman stated: "This sparked a gold rush, with people raising huge amounts of capital to build ultra-large data centers, but it also drove innovation on the edge side. People are trying to figure out how to fit a whole container of liquid into a very small cup, and the driving factor behind this is that current computing resources are not the main issue; bandwidth is. That's why RAM has become so expensive. This in turn has prompted a large number of researchers to find out what the key issues are and what we can do. This has led to innovations in quantification—reducing activation values and weights—to make models smaller while still maintaining their output capabilities."

The AI models come in various forms and sizes to accommodate a wide range of edge devices, from headphones and mobile phones to home appliances, industrial Internet of Things (IoT) systems, robots, and automotive systems. Designers need to choose a model with an appropriate number of parameters, as a model that is too large may slow down the device or even cause overheating. Conversely, if the model is too small, the product may not be intelligent enough.

The first step in matching the AI model for embedded applications is to define its purpose. Plowman states: "You must start from the perspective of what you want the model to do, but we can also use a million and one ways to make it smaller. Quantization and self-compression are both very effective methods. Distillation is another approach, and it usually yields better results on Transformer than on traditional convolutional neural networks (CNN). All these methods are coming together, making the gold rush for 'larger and stronger GPU' models less prevalent. Enabling more intelligent models to run on real embedded platforms is becoming increasingly feasible."

Figure 1: Common compression methods for the AI model

The size of the model must be taken into consideration along with the system as a whole. Imagimob, the Chief Technology Officer and Chief Product Officer of a company under Infineon Technologies, stated: "The challenge lies in designing a model that is complex enough to solve the problem, yet small enough to fit within the device. In addition, the model usually has to share memory and computing power with other parts of the application, so the available resources are actually less than what is indicated in the specification sheet. Optimization techniques such as pruning and quantization can reduce the size of the model, but they are typically applied after the design phase, often after training as well, and this comes at the cost of some accuracy. You need to choose an initial architecture of the right size from the beginning so that the optimized version can still fit within the device and meet the performance requirements. Aligning these three aspects—problem suitability, appropriate size, and accuracy after optimization—is where most of the work lies."

Others also believe that system-level efficiency is a core challenge. Rambus Outstanding inventor and researcher Steve Woo stated: "Edge AI is driving a shift from 'the largest model wins' to 'experience per watt.' Technologies such as compression, pruning, and low-precision arithmetic help to reduce the size of models to fit local resources, but deployment remains difficult, as memory capacity and bandwidth are often the limiting factors. Putting the model on a device is just the beginning. How to run it efficiently within strict power consumption and latency budgets is where the real engineering challenges begin."

If the wrong AI model is chosen, it may lead to additional work after deployment or result in poor product performance. Gedda says, "Once you have an optimized model, deployment is not actually that difficult. The model is converted into C code, compiled for the target platform, and then written to the device. The real challenge lies in fully leveraging the hardware's potential to achieve both energy-efficient and low-latency inference. There is an extremely wide variety of edge hardware – MCU, accelerators, NPU, DSP – and the standards are not yet well-established. Therefore, without the right tools, you will either end up with a general-purpose but inefficient deployment or have to do a lot of manual work for each target platform."

These deployment constraints make model selection a task that is highly relevant to the specific application. The appropriate architecture depends on the functions that the device must perform, as well as the workload it needs to support.

Cadence, the Senior Director of Chip Product Marketing, pointed out in a recent blog post: "Right-sizing inference for physical AI is of paramount importance. There is no one-size-fits-all computing configuration for intelligent physical systems. Some workloads require constant online, low-power inference; others need higher-performance multimodal processing across cameras, radar, lidar, audio, environmental sensors, and system status data."

Each hardware function and terminal device characteristic needs to be considered and defined. Sathish Balasubramanian, the product leader for Siemens' EDA EDA AI, stated: "One of the best technologies for matching the right models for edge AI is the ability to categorize functions, so that you don't need a large model. A model with 4 billion parameters is already quite large when placed on a sensor, even though we are now talking about 1 trillion parameters in the cloud. It is feasible to compress this model to less than 1 billion parameters and deploy it in each sensor. We need to define the functions with the highest priority that must be available, and train them well within the SLM (small language model). It can even be a ML (machine learning) model or a neural network model; it doesn't have to necessarily be a LLM."

After defining the necessary functions, designers can begin to build the core of AI, and then gradually add more features, including those that are essential, key, or additional enhancements.

Balasubramanian said, "From an architectural perspective, I observe that people typically define these three types of functions and how they enable intelligence. Once you have identified these categories, the essential functions must be as close to the sensors as possible. You can't make everything a mandatory requirement. You really need to prioritize them and design the system in a way that allows it to operate within a self-contained sensor loop, which means local computing, local memory requirements, and sometimes even a local backup environment. Everything needs to be separated accordingly. You also need at least one or two hours of power margin so that the device won't slow down or shut down."

In the automotive industry, companies are increasingly using the AI model for specific tasks and functions, yet the industry still largely relies on the CNN from three to four years ago. Plowman notes: "They have not fully transitioned to Transformer nor adopted the latest compression technologies. However, some external organizations have already launched image segmentation and object recognition systems that utilize Transformer."

For the IP suppliers responsible for optimizing these models, one challenge is that they may not have access to the models actually used by customers. Plowman says, "Automobile manufacturers train their own models, and for reasons of security, confidentiality, and proprietary IP, they keep these models very secret. We don't get to see them. We do receive a lot of proxy information about their appearance, or the proxy functions that they want us to accelerate, but we rarely have access to the entire models for study."

Figure 2: The IP modules of the edge AI and the physical AI. Source: Synopsys

When performing right-sizing, each IP module needs to be balanced with the overall model size. Synopsys, the Senior Director of Product Management, stated: "Right-sizing models for edge devices is a systemic issue, not a single-point optimization. Techniques such as quantization can help fit the models, but real-time latency, power consumption, cost, and functional safety all need to be balanced together. As the evolution speed of AI models exceeds the chip development cycle, the system also needs to have sufficient flexibility to support not only today's workloads but also the next generation of models."

Once the model is aligned with the system architecture, designers must determine which compression techniques can meet these constraints without sacrificing critical accuracy or performance.

Challenges of various compression technologies

Designers usually use a variety of compression techniques to resize models, and each method has its advantages and disadvantages.

For example, distillation can significantly reduce the size of a model, but it may also lead to more losses. Plowman says: “You will examine the model to identify which parts are being activated. You will do something similar to cutting out certain elements, but there will always be some loss in the teacher-student model. The student model tries to extract the core content that the teacher model knows from the larger model. This is why it may result in more inherent losses, but you will retain more reasoning capabilities. It can greatly reduce the size of the model, but if no further quantization is done, the options available to you are limited.”

Quantization causes the model to reach the goal in a more indirect manner, rather than moving in a straight line. Plowman says: “When input is applied to the network, there is a starting point. Then, depending on the values you provide, you need to navigate to a destination point. With each layer passed through, you are steering the transformation in a slightly different direction. When using full-precision values such as FP32 or FP16, you will reach a certain point more directly.”

Full-precision values fly in a straight line, while quantization is more like navigating through urban blocks. Plowman says: “We take one step upward, then one step to the right, another step upward, and then another step to the right, but in the end, you will end up at exactly the same position. The trick with quantization is to make these smaller, more angular steps convey the same meaning as direct navigation, while avoiding the power consumption, memory usage, and other costs associated with the relevant mathematical calculations. It starts by taking all the weights in the model and then putting them into buckets that match the quantization scheme. There are many, many different methods of quantization. This is a rather crude approach.”

A more gentle approach is self-compression. Plowman explains, “Self-compression technology recognizes that during the model training process, some paths become very popular, while others are less so. You never know what the final shape of these dimensions will be; it’s random with each retraining. You can manipulate this space to ensure that certain desired shapes are reproduced each time, or at least come close to them, but there is still a element of randomness involved. Some paths will emerge and serve as a framework for the rest of the system, but in the end, they are not particularly important. We then analyze the sensitivity of these paths, quantifying them over and over again until they contribute zero bits. If the model doesn’t change at all, yet you still get the same results, then the contribution of these paths is likely to be very small, or even negligible.”

Figure 3: The quantified Transformer architecture. Source: Imagination

For quantization and self-compression, the complexity lies in how to manage losses. Plowman points out: "You need to ensure that you don't lose what is truly necessary, and that there is sufficient good data coverage in the training set to support the quantization process."

In the end, successful edge AI deployments begin with defining the workload and selecting an architecture of the right size, rather than simply scaling down a pre-existing largest model. Compression can help to bridge the remaining gap, but its value depends on whether it is possible to maintain accuracy while meeting requirements for device power consumption, memory, latency, cost, and security. As models and hardware evolve at different rates, a flexible system design will become key to supporting current applications and future workloads.

Reference materials

• Structured vs Unstructured Pruning: An Exponential Gap (University of the Blue Coast, King's College London)
• "Self-Compressing Neural Networks" ( Imagination Technologies )

Can Edge AI Keep Up As the evolution speed of models exceeds that of chip cycles, experts are balancing to what extent architects can maintain adaptability without sacrificing power consumption, area, or efficiency.

Liz Allan