Image of an arrow

Donner une seconde jeunesse aux anciens SBC avec Linux en Temps Réel (Partie 1)

Avatar

dweiten

[L’introduction est en français, le reste du texte en anglais]

Linux avec l’option PREEMPT_RT peut permettre d’exécuter un kernel moderne et robuste sur des plateformes matérielles anciennes et aux performances limitées. Dans le cadre d’un parc matérielle en service, ces briques logicielles peuvent vous permettre d’éviter une coûteuse campagne de rappel. Ce sujet sera traité et décrit en deux parties. Le premier article aborde l’implémentation générique d’un Linux en RT ainsi que la configuration du noyau. La seconde partie montre les performances atteintes dans le cas particulier d’une plateforme datant de la dernière décennie. 😉

Introduction

SBCs and Linux have both evolved considerably. Today’s hardware far exceeds the capabilities available a few years ago. Linux now has better structure, is more secure, and is more feature-rich than ever before, but it is also more demanding of processing power.

Updating an older Linux system to a more modern Linux is highly desirable. Thousands of security vulnerabilities have been eliminated from the modern kernel and its associated programs and utilities. Modern Linux is also more stable and easier to configure.

Unable to Run Modern Linux?

Unfortunately, the increasing sophistication also comes a much higher computational demand. An SBC that is more than a few years old may not have the ability to meet performance requirements when running a modern Linux.

For instance, a system may be used to perform a simple control task on a specific schedule, along with background data reporting and communications functions. Although a particular older SBC may function satisfactorily with an older, smaller and less feature-rich kernel, when a more modern Linux system is installed, the extra overhead may cause the control task to miss deadlines, resulting in unsatisfactory function.

Non-essential features can be removed, restoring some of the performance losses. However, those gains are generally limited, and often even the most basic system will fail to meet requirements. Linux is just generally larger and more compute intensive than in the past.

Solving the Problem with Real-Time Linux

Savoir-faire Linux (SFL) recently had a customer with precisely this problem. They wanted to upgrade to a modern Linux for security and stability reasons, but when a recent Linux was installed, the system failed to meet performance requirements. It seemed that the only alternative might be to perform an SBC field replacement program.

Rather than upgrade the hardware of hundreds of field units, which would represent a significant expense and logistical challenge, SFL proposed that Real-Time Linux could bring performance back in line with their needs.

In order to demonstrate the improvement that can be expected, SFL took a very similar system and kernel, enabled Linux real-time extensions, and performed some tests. The results were interesting, and shall be discussed here.

What is Real-Time?

Real-time programming is widely talked about, but often misunderstood. Real-time does not generally mean “faster”, it means “bounded latency”, where the system will ensure that particular events will happen within certain (reasonable) bounds. This concept is generally understood by “bare-metal” microcomputer developers – the system must respond to certain stimuli in an appropriate maximum time, or system operation will fail.

Past Commercial Offerings

Many years ago, it was generally assumed that the Linux operating system was not appropriate for real-time applications. Commercial real-time alternatives such as QNX™ and VxWorks™ provided the performance required. Some open-source RTOSes existed, but none had the power and platform support that Linux possessed.

Early Real-Time Linux

Early efforts to provide real-time operation included RTAI and RTLinux, which provided bolt-on support for real-time applications with kernel patches and special programming paradigms to implement real-time processes. These did not implement real-time operation of actual Linux processes, making it difficult to implement real-time on existing non-real-time systems, or transparently mixing real-time and non-real-time applications.

The PREEMPT_RT Kernel Patch

The PREEMPT_RT kernel patch was developed as an alternative to provide Linux with real-time capability. It was developed over 15 years ago, and has since been incorporated by degrees into various distributions’ patch sets, then becoming an official Linux Foundation initiative called Real-Time Linux. The patch is now an official part of the Linux kernel, meaning that it can be enabled by configuration at kernel build time.

Linux Distribution Support for PREEMPT_RT

The support of PREEMPT_RT varies among Linux distributions. Most Linux distributionsare general-purpose operating systems, most often optimized for desktop, laptop, or mainframe use – and these applications do not generally require true real-time. There are always exceptions, of course. Financial processing systems, music processing systems, and others, sometimes rely on the determinism that PREEMPT_RT provides.

By default, most distributions do not enable PREEMPT_RT in their default kernels – generally, they must be sought out and explicitly installed. In fact, Ubuntu explicitly avoids a true real-time implementation because of potential security implications. In most distributions, PREEMPT_RT requires the selection of a new kernel in its package downloading program, but sometimes it might require full compilation of the kernel from source.

Use a Distribution in an Embedded System?

On the other hand, embedded systems do not generally use a Linux distribution. These devices are generally fixed-purpose with limited resources. It is best to use an embedded distribution creation tool, such as Yocto or Buildroot, which manage the source download, toolchain configuration, executable building, and image construction. They can also manage very reliable system upgrade. Moreover, in these tools, enabling PREEMPT_RT is generally as easy as turning on the option in the kernel configuration!

If an embedded distribution creation tool cannot be used, then the best approach may be to use one of the base level distributions that are highly customizable, such as Debian or Gentoo.

Implementation of Real-Time Linux

Kernel Configuration

Real-Time Linux is a kernel compiled with the PREEMPT_RT option enabled. Obtain the kernel source and configure it for your target system. Then, under General SetupPreemption Model, select Fully Preemptible Kernel (RT). Behind the scenes, this will also modify some other options that are required for the PREEMPT_RT option.

Once the PREEMPT_RT option is enabled, only minimal processing happens in the interrupt context, turning most of the work over to process threads (kind of like mini programs), which turns processing back to the kernel, ready for higher priority and real-time tasks. This allows the bulk of the processing for interrupts to be scheduled like any other process.

If these things are all that are changed, the improvement will be minor. To take full advantage, the programs run in user space must be modified as well.

Process Scheduler

The standard process schedule policy of SCHED_OTHER is generally be used for all Linux processes. Even in a real-time system, SCHED_OTHER is generally fine for all non-real-time processes.

Processes the require real-time will change their schedule policy to SCHED_FIFO or SCHED_RR, and set their rt_priority level. Raising a process’s priority generally is a “privileged” operation, so it may be necessary for the process to be run as root, or otherwise obtain root privilege.

Process Priorities

In the original Linux scheduling model, there are no true process priorities. However, Linux has the concept of “niceness”, which can be set from ‑20 (not very nice, takes more CPU time) to +19 (very nice, takes less CPU time).

With Real-Time Linux, it’s useful to consider each process having a priority (PR), between ‑100 (highest priority) and +40 (lowest priority).

A non-Real-Time Linux process, will still operate under SCHED_OTHER scheduling, but as the kernel is now fully preemptive under the hood, its niceness now sets its priority: PR=20+niceness. This means that non-Real-Time Linux process has PR between +1 to +40, at the top end of the overall priority range.

A Real-Time Linux process, compiled to use Real-Time scheduling, has a rt_priority from 0 (lowest priority) to +99 (highest priority)**, where the resultant PR=‑1‑rt_priority. Any Real-Time process has therefore a higher priority than all non-Real-Time processes, and there is also a priority order among Real-Time processes.

** In practice, the Real-Time priority limits might vary between operating systems and implementations, so it is good practice to call sched_get_priority_min() and sched_get_priority_max() to find the limits of settings on the local system.

Memory Allocation

Depending on the processor architecture and system load, locking memory pages into RAM will provide significant improvements in memory performance. A call to mlockall() can accomplish this, but there may be limitations unless the process is privileged, refer to the documentation for details.

Allocated areas may still experience an initial delay while being locked into RAM on their first access. It is good practice to fully initialize every allocated memory area, creating a “pre‑fault” and loading the MMU for rapid future access.

Other Kernel Build Optimizations

Precise Timing

If precise timing is a requirement, then also enable the HIGH_RES_TIMERS option, and increasing the “system tick” rate from the default to a higher value, perhaps to 500 Hz. This makes general operation and timing of the system more precise, at a small extra overhead.

The HIGH_RES_TIMERS option is also required for precise time measurement in function traces, mentioned later.

For precise delay timing, it is important to use nanosleep() or clock_nanosleep() in conjunction with the HIGH_RES_TIMERS option. Even in the presence of high resolution timing, usleep() will give relatively poor results. Refer to the nanosleep(2) manpage.

Function Tracing

FTrace is an extremely useful tool for performance measurement of a Linux system. In order to use it, the kernel is compiled with the CONFIG_FTrace enabled. It is also possible to enable tracing many events, such as preempt activities, scheduling latency, hardware latency, block IO transactions and GPIO events.

For background on tracing, LWN.net has several excellent articles: A look at FTrace, Debugging the kernel using FTrace part 1 and part 2, and Secrets of the FTrace function tracer.

The article trace-cmd: A front-end for FTrace is also interesting, although in our case, its trace start/stop function is performed internally by our test application itself.

Build for Speed

The kernel has an option to optimize compilation for size or for speed. The default is for speed, but it would be appropriate to check to ensure that CC_OPTIMIZE_FOR_PERFORMANCE is enabled.

Summary

It may not be possible to run a modern Linux on an older system, because the increased burden would prevent specific operations from occurring on time. Using a modern Linux kernel with the PREEMPT_RT patch, and performing time-sensitive operations in Real-Time-enabled processes, it is possible to meet performance requirements on an old system.

The next article will consider a particular system implementation under experimental conditions, and bring up interesting aspects of the performance and operation of such a system using Real-Time Linux.

 

  1. This is a good article about Linux Preempt-RT, and it is a good solution to have an application with a high priority to access to the hardware.
    I uses this solution to manage a music player with a high priority to have a perfect sound, and a control application which may accept latency even if it use a lot of resources, like a voice recognizer.
    But can you explain your comparison with RTOS ? You compare system for MCU without MMU with Linux which need a CPU and MMU. Even if you use old platforms with small resources, the hardware is totally different and they are applied for different kinds of usage.
    It should be good to have a graphic about the latency of the process and the hardware interruptions with Preempt-RT.

    1. Thank you for your comment. Yes, Preempt-RT is often used for music. I should have mentioned this.

      The comparison to other RTOSes was to point out that there have long been commercial solutions for the need for real-time processing. However, they can be expensive, and do not have the universal reach that Linux can provide. I was not thinking of the non-MMU world at all, but I was thinking of, for example, QNX and VxWorks, which are often used in similar CPUs that do have MMUs (e.g. X86, MIPS and some ARM). You are absolutely correct, we need to discuss apples-to-apples. A non-MMU system is a completely different challenge, and well beyond the scope of this article.

      If you used to run an old Linux system, say circa Y2K, and you want to update to a more modern Linux system, you might run into performance issues. By using Preempt-RT and assigning RT priority to some processes, deadlines can continue to be met. Of course, it will always require analysis and careful design.

      The next article should provide more detail of a specific case study of doing exactly this – replacing an old Linux system with Preempt-RT, and some analysis of a kernel with Preempt-RT versus one without.

Laisser un commentaire

Votre adresse courriel ne sera pas publiée. Les champs obligatoires sont indiqués avec *

Ce site utilise Akismet pour réduire le pourriel. En savoir plus sur comment les données de vos commentaires sont utilisées.


Articles similaires

Image of an arrow

[L’introduction est en français, le reste du texte en anglais] Le premier article de cette série présentait les motivations et les avantages de l’intégration d’une image de conteneur dans un système Linux embarqué et examinait les défis que ce nouveau paradigme soulève. Le second article présentait différentes approches pour intégrer un conteneur dans le système […]

Thumbnail image

[L’introduction est en français, le reste du texte en anglais] Points clés: La série de correctifs PREEMPT_RT permet au noyau Linux de répondre aux besoins des systèmes temps-réel. Différents réglages permettent un compromis entre performance et latence. Un paquet de support de carte minimal permet une réduction de la latence. Les mesures de durcissement du […]

Thumbnail image

[L’introduction est en français, le reste du texte en anglais] Les produits industriels deviennent de plus en plus complexes et nécessitent de beaucoup plus d’interaction et de connectivité. Il est maintenant usuel de faire le suivi et la maintenance d’un appareil à partir d’une interface par une application web embarqué sur ce même appareil. Au […]

Précuseur de l’Internet post-GAFAM             Las Vegas, le 9 janvier 2019 ​ – Présente au Consumer Electronics Show de Las Vegas, Savoir-faire Linux, entreprise québécoise de services-conseils depuis plus de 20 ans et experte dans l’industrie du logiciel libre au Canada, lance l’application de communication universelle et autonome Jami. Destiné […]

Rennes, le 30 novembre 2017 – Savoir-faire Linux est fière d’annoncer l’installation à Rennes d’un centre d’expertise en ingénierie logicielle libre et systèmes embarqués pour desservir ses clients français et européens. Savoir-faire Linux est une entreprise québécoise experte en technologies libres et open source, leader au Canada et implantée en France depuis 2014. Elle a […]