Image of an arrow

Small language models on i.MX95: LFM at the Edge

Avatar

tgarbay

Small Language Models (SLMs) on the i.MX95

LFM3-1.7B running offline on the i.MX95 CPU in real time: first text in about 3 s, then about 5 tokens/s.

In this article, we explore the capabilities of the i.MX95 for running SLMs. We already have developed several applications on the i.MX95, including monocular depth estimation.

Small language models on i.MX95 required more than a language interface on an embedded device that load a model. It must accept input, produce a useful answer within an acceptable delay, coexist with the rest of the system, and remain available without a cloud connection. These requirements make hardware measurements and application architecture as important as model selection.

We investigated these questions on the NXP FRDM i.MX95 using Liquid AI language models, llama.cpp, and a Yocto-based Linux environment. We considered two interaction paths: a local text interface and spoken questions answered on a local screen.

The measurements show that for small language models on i.MX95 both paths are practical on the board’s CPU alone. A 230M-parameter model starts answering a short text question in 0.28 seconds, and an audio model starts answering a four-second spoken question 2.4 seconds after the recording ends. GPU-assisted execution is slower in every tested configuration, but it reduces average CPU utilization from more than 90% to less than 10%. This frees processor capacity for the rest of the application.

Applications: useful assistance without a cloud dependency

Small language models on i.MX95 pave the way for many applications. Industrial maintenance assistance is one application: a technician standing beside a machine could ask for an explanation of a documented alarm or for the relevant inspection procedure. A local interface keeps every interaction independent of network availability on the shop floor.

Embedded documentation is a related but distinct use case. A device could provide conversational access to manuals stored locally, rather than requiring users to navigate a large document hierarchy. Retrieval-augmented generation (RAG) is a natural extension: relevant passages are retrieved, inserted into the model context, and used to ground the response. The NXP eIQ GenAI Flow provides such a retrieval layer and could be combined with the system presented here.

Voice input makes these interfaces more accessible when typing is inconvenient, for example when wearing gloves or handling tools. Our audio path targets a spoken question followed by a written answer. Autonomous kiosks and local terminals offer another setting: users ask short questions about a bounded service or procedure, with models and content stored on the device. Here, response consistency, recovery after interruption, and a clear interface may matter more than open-ended conversation.

In all of these cases, inference runs entirely on the device: questions and recordings do not need to leave it.

Hardware, software, and model selection

For small language models on i.MX95, the evaluation platform is the FRDM i.MX95, with six Arm Cortex-A55 CPU cores and an Arm Mali-G310 GPU. The system exposes approximately 7.5 GiB of RAM to Linux.

small language models on i.MX95

FRDM i.MX 95 (https://www.nxp.com/design/design-center/development-boards-and-designs/FRDM-IMX95)

We use a Yocto-based image with the NXP board support package, local model files, a USB keyboard, a USB headset microphone, and an HDMI display path. The CPU backend uses six threads. The GPU-assisted backend uses Vulkan. It does not mean that every operation executes on the GPU. The i.MX95’s Neutron NPU is not used in these experiments.

The main comparison includes four Liquid Foundation Model (LFM) from Liquid AI:

InteractionModelLanguage-model weightsAudio encoder/projector
TextLFM2.5-230MQ4_K_MNot applicable
TextLFM3-1.7BQ4_K_MNot applicable
Audio input, text outputLFM2.5-Audio-1.5BQ4_0Q4_0
Audio input, text outputLFM3-Audio-2BQ4_0Q8_0

The weights are packaged as GGUF files. Q4_K_M and Q4_0 are quantization formats intended to reduce weight storage, not statements that every tensor or computation uses four bits. Runtime memory also includes context state, intermediate buffers, and backend allocations.

Model names describe substantially different parameter counts: this is not an equal-size comparison between model generations. The different audio-projector precisions introduce another variable. The results characterize deployment configurations, not an isolated architectural advantage of LFM2.5 or LFM3.

Earlier work also examined LFM2.5-350M and LFM2.5-1.2B-Instruct. Those measurements remain useful background, but they are not part of the four-model comparison plotted below.

Text application: from keyboard input to local output

small language models on i.MX95

Functional architecture of the local text application. The blocks represent processing stages.

First system with small language models on i.MX95 is text processing. The keyboard supplies a prompt through the terminal to llama-cli. Before inference, the application applies the model’s chat template: user, assistant, and system messages must have the expected structure and separators. The resulting conversation is then tokenized into numerical identifiers.

Inference has two distinct phases. Prefill processes the input context and prepares the model state. Autoregressive decoding then generates the answer one token at a time. Detokenization converts those tokens into readable fragments, which are written progressively to standard output.

Keeping the CLI process alive avoids reloading the model for every question. Conversation history can be retained, although longer histories increase the context that must be managed.

The local interface uses tmux and xterm to present the conversation, examples, and monitoring information. Xwayland, Weston, and the Linux DRM/KMS display stack provide the path to the HDMI screen. These interface components are part of the application workload, and their resource requirements cannot be inferred from an isolated model benchmark.

Audio application: capture, encode, and answer in text

small language models on i.MX95

Target audio integration around a persistent CLI. The illustrated encoder runs on CPU, the GPU benchmark configuration additionally requests projector offload.

Second system with small language models on i.MX95 is audio processing. The microphone path uses ALSA, PipeWire, and pw-record to capture mono, 16 kHz, PCM16 audio into a temporary WAV file. Recording finishes before inference starts.

The encoder transforms the waveform into learned audio representations. A projector adapts those representations to the language model’s input space. Audio embeddings and tokenized system instructions then drive generation.

For the target integration, a controller keeps llama-cli running, stages each WAV, and reads the resulting text stream. Clearing conversation history between questions preserves the existing independent-question behavior without unloading the model.

Measuring a defined workload

All measurements use llama.cpp build b11960 with a 4,096-token context, one parallel slot, a logical batch size of 256, and a micro-batch size of 64. CPU execution disables device offload. Vulkan execution requests all language-model layers and, for audio, the projector on the Mali-G310. Models are already loaded, and prompts are not cached. Each configuration is run five times after one excluded warm-up run; the figures report medians with observed minimum-to-maximum ranges, not confidence intervals. Time to first text (TTFT) runs from the start of the request to the first non-whitespace response text.

Text requests contain exactly 32 or 128 input tokens, including chat framing and system instructions. Longer inputs were also tested but are outside the scope of the targeted applications, for which 128 tokens comfortably covers a user question. Here are examples of 32-token and 128-token texts:

32-tokens about Edge AI:

Edge AI runs machine learning models directly on local devices, processing sensor data without relying on cloud services. It supports offline operation and keeps information close to its source.

128-tokens about Edge AI:

Edge AI brings machine learning to devices that collect and use data, such as industrial gateways, cameras, and local terminals. Instead of sending every input to a remote service, the device runs a model on its own processor. This can support operation without an internet connection and reduce the amount of data transferred. Practical deployment on real hardware still requires careful choices: model size, memory use, response time, power consumption, and answer quality must match the application. CPU, GPU, and dedicated accelerators offer different tradeoffs. For a maintenance assistant, reliable access to local documentation and clear handling of uncertain answers matter as much as raw inference speed.

Text results: prompt length and model size both matter

small language models on i.MX95

Time to first text and tokens per second for each prompt length.

LFM2.5-230M reaches first text in 0.28 seconds on CPU for a 32-token input, and in 0.95 seconds for 128 tokens. It then generates about 33 tokens per second (32.82 and 32.96, respectively). The Vulkan path takes 0.93 and 1.97 seconds to start, then generates approximately 7.4 tokens per second.

LFM3-1.7B operates in a different envelope. On CPU, TTFT rises from 2.37 seconds at 32 tokens to 9.17 seconds at 128, with generation at about 5.9 tokens per second. GPU-assisted execution takes 7.34 and 15.26 seconds before first text, then generates approximately 1.2 tokens per second.

Generation speed is easier to interpret against reading speed. Adults read English silently at about four words per second on average (Brysbaert, 2019), which corresponds to roughly five tokens per second. On CPU, LFM3-1.7B therefore produces text at about reading pace, and LFM2.5-230M about six times faster. At around 1.2 tokens per second, the GPU-assisted LFM3-1.7B writes noticeably slower than a user reads.

Where the time goes: input length versus GPU overhead

The two prompt lengths also reveal where time is spent. On CPU, the 96 additional input tokens add 0.67 seconds for LFM2.5-230M and 6.8 seconds for LFM3-1.7B, an incremental prefill rate of roughly 140 and 14 tokens per second. On the GPU-assisted path, the same 96 tokens add 7.9 seconds for LFM3-1.7B, about 12 tokens per second, close to the CPU rate. Most of the GPU penalty for this model therefore comes from a large component that does not depend on prompt length, together with decoding that is about five times slower than on CPU. As a result, the gap in TTFT between GPU and CPU narrows from 3.1 times at 32 tokens to 1.7 times at 128 tokens for this model.

These values make input construction a product concern. System instructions, conversation history, and retrieved documentation all consume the same context budget. A short visible question does not guarantee a short model input: with LFM3-1.7B on CPU, each additional 100 tokens of context adds about seven seconds before the first word appears.

Audio results: distinguish recording time from processing time

small language models on i.MX95

Spoken question to written answer: on CPU, the wait is comparable to the question itself

For LFM2.5-Audio-1.5B, CPU TTFT is 2.40 seconds, followed by generation at 10.72 tokens per second. GPU-assisted execution takes 13.50 seconds before first text and generates 2.41 tokens per second.

LFM3-Audio-2B reaches first text in 4.25 seconds on CPU, with generation at 6.91 tokens per second. The GPU-assisted path takes 22.46 seconds and generates 1.61 tokens per second.

Relative to the 4.065-second question, LFM2.5-Audio-1.5B on CPU starts answering after about 0.6 times the question’s duration, and LFM3-Audio-2B after roughly the duration of the question itself. Both then generate text faster than reading pace. For a spoken interface, this means the processing wait is comparable to the time the user has just spent speaking.

For this workload, CPU execution is the stronger starting point when responsiveness is the priority: GPU-assisted TTFT is five to six times longer. These results do not establish a general ordering across all audio tasks, particularly because model sizes and projector quantizations differ.

GPU offload trades responsiveness for CPU availability

small language models on i.MX95

Trade-off: slower first text on the GPU path, but most of the CPU left free.

The figure above shows why a slower backend can still be worth considering. In CPU mode, inference keeps the processor almost fully busy for the whole request, between 90% and 97% of the six cores in every configuration. With GPU assistance, utilization drops to between 5% and 9%, or about half a core at most on average.

For LFM3-1.7B at 128 input tokens, CPU utilization falls from 97.1% to 6.1%, while TTFT increases from 9.17 to 15.26 seconds. For LFM2.5-Audio-1.5B, utilization falls from 95.8% to 5.5%, while TTFT rises from 2.40 to 13.50 seconds. LFM3-Audio-2B shows the same pattern: 96.4% versus 6.0% CPU utilization, and 4.25 versus 22.46 seconds before first text.

These are board-wide measurements from CPU counters, normalized across the six cores. They cover the full request, including audio upload where applicable and text generation, not only the interval before first text. Because GPU-assisted requests last longer, the lower average does not translate directly into proportionally less total CPU work. What it does guarantee is that, while the model is running, most of the CPU remains available to other tasks.

This matters for devices where the language interface is one function among several, for example alongside sensor acquisition, communication, or control loops that must keep their timing. In that situation, a slower but predictable assistant that leaves the CPU largely free can be preferable to a fast one that saturates it.

Engineering the complete system

Our contribution is the integration between the model runtime and the embedded platform: a reproducible Yocto environment, board-level interfaces, CPU and Vulkan execution paths, application lifecycle management, and measurements with explicit boundaries.

That integration requires more than selecting a GGUF file. Model revisions, checksums, runtime versions, and launch parameters must be traceable. A model process should not be reloaded unnecessarily, and failed input must not leave the application or accelerator in an unusable state.

Operational requirements are equally important: startup and recovery, bounded recording and output lengths, temporary-file cleanup, update compatibility, and model licensing. Offline execution limits the need to transmit user input, but it does not by itself define how recordings, prompts, or logs are retained and protected.

A measured basis for choosing an architecture

The i.MX95 runs the tested text and audio-to-text workloads locally, without a cloud connection and without the NPU. On CPU, a small text model answers in well under a second and generates text several times faster than a user reads, a 1.7B model generates at about reading pace, and an audio model starts answering a short spoken question in less time than it took to ask it.

The measurements translate into a few practical starting points:

NeedStarting configurationMeasured behavior
Short questions on a bounded topic (kiosk, FAQ)LFM2.5-230M on CPU0.28–0.95 s to first text, about 33 tokens/s
Richer answers, documentation assistanceLFM3-1.7B on CPU2.4–9.2 s to first text, about 6 tokens/s
Spoken questions, hands-busy usersLFM2.5-Audio-1.5B on CPU2.4 s to first text after a 4-second question
Assistant alongside CPU-intensive or time-critical tasksGPU-assisted (Vulkan) executionUnder 10% average CPU utilization, at 1.7 to 5.6 times the TTFT

Model size, context length, and response length remain the main design variables, and answer quality must be validated on the target task. The choice between CPU and GPU-assisted execution is not a question of which is faster, but of which resource the rest of the application needs most.

Conclusion

The i.MX95 can host a useful language interface without a cloud connection and without the NPU. GPU-assisted execution is slower in every tested configuration, but it keeps average CPU utilization below 10%. That makes it a reasonable option when the language interface must share the board with acquisition, communication, or control tasks.

The deciding factors are therefore not only the model, but also the context budget, the response length, and which resource the rest of the application needs most.

Key takeaways

  1. CPU inference is responsive enough for interactive use. On the six Cortex-A55 cores, a 230M model gives sub-second first text at about 33 tokens per second, and a 1.5B audio model starts answering a spoken question in less time than it took to ask it.
  2. Input length is a product decision. With LFM3-1.7B on CPU, every additional 100 tokens of context (system instructions, conversation history, or retrieved documentation) adds about seven seconds before the first word appears.
  3. GPU offload trades latency for CPU availability. Vulkan execution on the Mali-G310 is 1.7 to 5.6 times slower to first text, but it lowers average CPU utilization from more than 90% to less than 10%. This helps when the assistant is one function among several on the device.

Leave a comment

Your email address will not be published. Required fields are marked *


Similar articles

Image of an arrow

AI coding agents can now modify repositories, run tests, and propose commits. Yet writing code is only one part of engineering work. Between a ticket and a merged change, several steps are still required: gathering context, preparing the environment, producing a patch, opening a review, waiting for feedback, and updating the code until it is […]

Prompt Anything: Instance Segmentation with YOLOE-26 within a Hailo-10H Running YOLOE-26 as a “prompt anything” segmentation within a edge NPU, from model surgery and INT8 quantization to a 15.6 FPS (640px) / ~31 FPS (320px) fully optimized live pipeline. YOLOE-26’s open-vocabulary support lets operators type any class name for real-time segmentation on the Hailo-10H. Our […]

Monocular Depth Estimation on the Verdin i.MX95 for Edge AI   Deploying Monocular Depth Estimation on the Verdin i.MX95 for Edge AI: model adaptation, INT8 quantization, and a full embedded software stack for real-time depth inference at 30 FPS from a single RGB camera. Monocular depth estimation involves reconstructing a depth map from a single […]

Live demonstrations at Booth 4-642 (Hall 4) will showcase real-time AI vision pipelines optimized for embedded hardware accelerators and built on the open-source AI ecosystem.   Nuremberg, Germany, March 6, 2026 – Savoir-faire Linux, a leading open-source software engineering and consulting firm specializing in embedded and industrial systems, announces at Embedded World 2026 the official launch […]