Key Takeaways: The first step in right-sizing an AI model for an embedded application is determining its purpose, including the target device functions and product features. Designers typically use a mix of compression techniques rather than a single method. Strategies include quantization, distillation, and pruning. Small language models (SLMs) and compact edge models require tight system-level integration to ensure optimal efficiency. Today’s AI industry faces two very different challenges. On one side, enormous AI data centers and AI factories are seeking more efficient ways to process high-parameter large language models (LLMs), with bandwidth and memory as the key bottlenecks. On the other side are smaller AI models running locally on edge devices, where there isn’t enough area, power, or memory for complex, higher-parameter LLMs. “This has led to a gold rush with people raising huge amounts of capital to build mega data centers, but has also sparked innovation on the edge,” said Ed Plowman, CTO at Imagination Technologies . “People are trying to work out how they can cram an entire container of liquid into a very small cup, and that has been driven by the fact that compute resources are not the main problem at the moment. It’s bandwidth, and that’s why RAM has gotten so expensive. That triggered a whole bunch of researchers to look at what the key problems are and what we can do about it. That triggered innovations around quantization — reducing activations and weights simultaneously — to make the models even smaller but still maintaining the output capability of the model.” AI models come in many shapes and sizes to fit a wide range of edge devices, from earbuds to phones, home appliances, industrial IoT, robotics, and automotive systems. Designers need to choose a model with the right number of parameters because an oversized model can slow the device down or cause it to overheat. On the flip side, if the model is too small, the product may not be smart enough. The first step in right-sizing an AI model for an embedded application is to define its purpose. “You have to start from the point of view of what you want the model to do, but then there are a million and one things we can throw at it to make it smaller,” said Plowman. “Quantization and self-compression are very effective tools. Distillation is another approach, and it works better on transformers than on traditional convolutional neural networks (CNNs). All these things are starting to combine to get us to a point where the gold rush for bigger and bigger GPUs is less prevalent. The enablement in embedded systems and the progression towards quite intelligent models being able to run on truly embedded platforms is becoming more realistic.” Technique How It Works Key Characteristics Distillation Trains a smaller student model to mimic a larger teacher. Transfers learned behavior into a smaller model. Pruning Removes low-impact weights, neurons, heads, or layers. Structured (whole units) or unstructured (individual weights). Quantization Uses fewer bits for weights and activations. Retains network structure; correction limits accuracy loss. Self-compression Gradually reduces precision, potentially to zero bits. Managed compression that can target proprietary hardware. Fig. 1: Common ways to compress AI model The model size must be considered alongside the system. “The hard part is designing a model that is sophisticated enough to solve the problem but small enough to fit the device,” said Magnus Gedda, CTO and chief product officer at Imagimob, an Infineon Technologies company. “On top of that, the model usually shares memory and compute with the rest of the application, so the available resources are smaller than the spec sheet suggests. Optimization techniques like pruning and quantization can shrink a model, but they are applied after design and usually after training, and they cost some accuracy. You need to pick an initial architecture that is the right size