Using Linux with the PREEMPT_RT option can make it possible to run a modern, secure kernel on an older, limited performance single board computer (SBC), perhaps avoiding a costly hardware upgrade campaign. This topic will be addressed in two parts. The first article will consider Real-Time Linux implementation in the kernel.

The second article will consider a particular case and, using a simple test application, shall provide some interesting data regarding the performance on a particular (rather old ☺) processor.

Introduction

SBCs and Linux have both evolved considerably. Today’s hardware far exceeds the capabilities available a few years ago. Linux now has better structure, is more secure, and is more feature-rich than ever before, but it is also more demanding of processing power.

Updating an older Linux system to a more modern Linux is highly desirable. Thousands of security vulnerabilities have been eliminated from the modern kernel and its associated programs and utilities. Modern Linux is also more stable and easier to configure.

Unable to Run Modern Linux?

Unfortunately, the increasing sophistication also comes a much higher computational demand. An SBC that is more than a few years old may not have the ability to meet performance requirements when running a modern Linux.

For instance, a system may be used to perform a simple control task on a specific schedule, along with background data reporting and communications functions. Although a particular older SBC may function satisfactorily with an older, smaller and less feature-rich kernel, when a more modern Linux system is installed, the extra overhead may cause the control task to miss deadlines, resulting in unsatisfactory function.

Non-essential features can be removed, restoring some of the performance losses. However, those gains are generally limited, and often even the most basic system will fail to meet requirements. Linux is just generally larger and more compute intensive than in the past.

Solving the Problem with Real-Time Linux

Savoir-faire Linux (SFL) recently had a customer with precisely this problem. They wanted to upgrade to a modern Linux for security and stability reasons, but when a recent Linux was installed, the system failed to meet performance requirements. It seemed that the only alternative might be to perform an SBC field replacement program.

Rather than upgrade the hardware of hundreds of field units, which would represent a significant expense and logistical challenge, SFL proposed that Real-Time Linux could bring performance back in line with their needs.

In order to demonstrate the improvement that can be expected, SFL took a very similar system and kernel, enabled Linux real-time extensions, and performed some tests. The results were interesting, and shall be discussed here.

What is Real-Time?

Real-time programming is widely talked about, but often misunderstood. Real-time does not generally mean “faster”, it means “bounded latency”, where the system will ensure that particular events will happen within certain (reasonable) bounds. This concept is generally understood by “bare-metal” microcomputer developers – the system must respond to certain stimuli in an appropriate maximum time, or system operation will fail.

Past Commercial Offerings

Many years ago, it was generally assumed that the Linux operating system was not appropriate for real-time applications. Commercial real-time alternatives such as QNX™ and VxWorks™ provided the performance required. Some open-source RTOSes existed, but none had the power and platform support that Linux possessed.

Early Real-Time Linux

Early efforts to provide real-time operation included RTAI and RTLinux, which provided bolt-on support for real-time applications with kernel patches and special programming paradigms to implement real-time processes. These did not implement real-time operation of actual Linux processes, making it difficult to implement real-time on existing non-real-time systems, or transparently mixing real-time and non-real-time applications.

The PREEMPT_RT Kernel Patch

The PREEMPT_RT kernel patch was developed as an alternative to provide Linux with real-time capability. It was developed over 15 years ago, and has since been incorporated by degrees into various distributions’ patch sets, then becoming an official Linux Foundation initiative called Real-Time Linux. The patch is now an official part of the Linux kernel, meaning that it can be enabled by configuration at kernel build time.

Linux Distribution Support for PREEMPT_RT

The support of PREEMPT_RT varies among Linux distributions. Most Linux distributionsare general-purpose operating systems, most often optimized for desktop, laptop, or mainframe use – and these applications do not generally require true real-time. There are always exceptions, of course. Financial processing systems, music processing systems, and others, sometimes rely on the determinism that PREEMPT_RT provides.

By default, most distributions do not enable PREEMPT_RT in their default kernels – generally, they must be sought out and explicitly installed. In fact, Ubuntu explicitly avoids a true real-time implementation because of potential security implications. In most distributions, PREEMPT_RT requires the selection of a new kernel in its package downloading program, but sometimes it might require full compilation of the kernel from source.

Use a Distribution in an Embedded System?

On the other hand, embedded systems do not generally use a Linux distribution. These devices are generally fixed-purpose with limited resources. It is best to use an embedded distribution creation tool, such as Yocto or Buildroot, which manage the source download, toolchain configuration, executable building, and image construction. They can also manage very reliable system upgrade. Moreover, in these tools, enabling PREEMPT_RT is generally as easy as turning on the option in the kernel configuration!

If an embedded distribution creation tool cannot be used, then the best approach may be to use one of the base level distributions that are highly customizable, such as Debian or Gentoo.

Implementation of Real-Time Linux

Kernel Configuration

Real-Time Linux is a kernel compiled with the PREEMPT_RT option enabled. Obtain the kernel source and configure it for your target system. Then, under General Setup → Preemption Model, select Fully Preemptible Kernel (RT). Behind the scenes, this will also modify some other options that are required for the PREEMPT_RT option.

Once the PREEMPT_RT option is enabled, only minimal processing happens in the interrupt context, turning most of the work over to process threads (kind of like mini programs), which turns processing back to the kernel, ready for higher priority and real-time tasks. This allows the bulk of the processing for interrupts to be scheduled like any other process.

If these things are all that are changed, the improvement will be minor. To take full advantage, the programs run in user space must be modified as well.

Process Scheduler

The standard process schedule policy of SCHED_OTHER is generally be used for all Linux processes. Even in a real-time system, SCHED_OTHER is generally fine for all non-real-time processes.

Processes the require real-time will change their schedule policy to SCHED_FIFO or SCHED_RR, and set their rt_priority level. Raising a process’s priority generally is a “privileged” operation, so it may be necessary for the process to be run as root, or otherwise obtain root privilege.

Process Priorities

In the original Linux scheduling model, there are no true process priorities. However, Linux has the concept of “niceness”, which can be set from ‑20 (not very nice, takes more CPU time) to +19 (very nice, takes less CPU time).

With Real-Time Linux, it’s useful to consider each process having a priority (PR), between ‑100 (highest priority) and +40 (lowest priority).

A non-Real-Time Linux process, will still operate under SCHED_OTHER scheduling, but as the kernel is now fully preemptive under the hood, its niceness now sets its priority: PR=20+niceness. This means that non-Real-Time Linux process has PR between +1 to +40, at the top end of the overall priority range.

A Real-Time Linux process, compiled to use Real-Time scheduling, has a rt_priority from 0 (lowest priority) to +99 (highest priority)**, where the resultant PR=‑1‑rt_priority. Any Real-Time process has therefore a higher priority than all non-Real-Time processes, and there is also a priority order among Real-Time processes.

** In practice, the Real-Time priority limits might vary between operating systems and implementations, so it is good practice to call sched_get_priority_min() and sched_get_priority_max() to find the limits of settings on the local system.

Memory Allocation

Depending on the processor architecture and system load, locking memory pages into RAM will provide significant improvements in memory performance. A call to mlockall() can accomplish this, but there may be limitations unless the process is privileged, refer to the documentation for details.

Allocated areas may still experience an initial delay while being locked into RAM on their first access. It is good practice to fully initialize every allocated memory area, creating a “pre‑fault” and loading the MMU for rapid future access.

Other Kernel Build Optimizations

Precise Timing

If precise timing is a requirement, then also enable the HIGH_RES_TIMERS option, and increasing the “system tick” rate from the default to a higher value, perhaps to 500 Hz. This makes general operation and timing of the system more precise, at a small extra overhead.

The HIGH_RES_TIMERS option is also required for precise time measurement in function traces, mentioned later.

For precise delay timing, it is important to use nanosleep() or clock_nanosleep() in conjunction with the HIGH_RES_TIMERS option. Even in the presence of high resolution timing, usleep() will give relatively poor results. Refer to the nanosleep(2) manpage.

Function Tracing

FTrace is an extremely useful tool for performance measurement of a Linux system. In order to use it, the kernel is compiled with the CONFIG_FTrace enabled. It is also possible to enable tracing many events, such as preempt activities, scheduling latency, hardware latency, block IO transactions and GPIO events.

For background on tracing, LWN.net has several excellent articles: A look at FTrace, Debugging the kernel using FTrace part 1 and part 2, and Secrets of the FTrace function tracer.

The article trace-cmd: A front-end for FTrace is also interesting, although in our case, its trace start/stop function is performed internally by our test application itself.

Build for Speed

The kernel has an option to optimize compilation for size or for speed. The default is for speed, but it would be appropriate to check to ensure that CC_OPTIMIZE_FOR_PERFORMANCE is enabled.

Summary

It may not be possible to run a modern Linux on an older system, because the increased burden would prevent specific operations from occurring on time. Using a modern Linux kernel with the PREEMPT_RT patch, and performing time-sensitive operations in Real-Time-enabled processes, it is possible to meet performance requirements on an old system.

The next article will consider a particular system implementation under experimental conditions, and bring up interesting aspects of the performance and operation of such a system using Real-Time Linux.