Containers on Linux embedded systems

The use of containers is not a new idea and nowadays several container solutions for Linux embedded systems are available such as BalenaOS, Linux microPlatform or Torizon from Toradex. These projects provide a Linux system with a Docker based infrastructure to run and manage containers along with many tools to provide an easy developer user experience.

Our goal in this series of articles is to understand and to provide an overview of the possible approaches to leverage containers benefits in a lightweight and minimal way regarding the Yocto integration while trying to understand, fulfill our requirements and make an informed choice about different technical approaches.

Three articles will dive into this problem from a global point of view down to Yocto, Docker and Podman technical aspects.

In this first article, the general motivations on running one or multiple containers in a Linux embedded system are exposed as long as the target system is not limited by any hardware design. However, this architecture brings up many challenges and open questions:

• some are tied to the embedded systems context, such as reliability, performance and speed,
• other are tied to the product engineering and commercialization such as traceability, reproducibilty and license compliance.

Some of the challenges raised in this article series are left open as they are topics for separate discussions.

In the next article, different technical approaches are explained based on the Yocto build system within our technical environment as the target architecture and the build requirements with their constraints. Following the final goal to embed a container image in the target read-only root filesystem, two approaches stand out: integrate a docker image archive or integrate a Docker store in the target root filesystem.

Finally, the different technical Yocto’s recipes will be detailed showing how the approaches mentioned above can be integrated in the Yocto build system.

Why running containers on embedded Linux is a good thing ?

Containers on embedded devices are quite easy to use and widespread in the embedded world. Using application containerization on embedded devices has many advantages and can solve some problems as architectural constraints, running legacy code in a mainstream platform, workload flexibility…

It can be needed when:

• You want to focus only on the application development regardless of the platform on which the container is running.Interfaces between the system and the container have to be defined as the container may need to access to material resources as audio, graphical devices or even storage resources such mounted devices or volumes for example.

• You want the application environment to be independent of the base system OS. For example, the application needs to install some specific distribution package or conflicting libraries which would generate incompatible dependencies. That way, the application container can easily be run on multiple platforms.
• You may want to have different development cycles regarding the base OS system and the application and want to update them separately.

Containerization has many benefits, here is a quick list of advantages:

• • userspace isolation of containers applications,
  • decouple application development from embedded Linux concerns,
  • really low overhead,
  • runs on constrained hardware,
  • brings platform modularity,
  • speed up development time.

Challenges

Despite the several benefits that are listed above, container integration into embedded systems brings up relevant questions that apply in the industrial and embedded world.

• Hardware resource limitations

Having one or multiple containers in embedded devices can be cumbersome. Sometimes, these devices do not always have enough storage capacity, hardware resources or RAM to run a fully distro-based container. The application container has to be designed taking into account these potential constraints.

Even if the containerization has nearly the same performance as an application in the native base system and does not add system overhead in most use cases, these constraints have to be taken into account during the product design.

• Improve system reliability and maintain data integrity

A major cause of embedded device failure is data corruption and it can be critical on root filesystems. Data corruption can cause arbitrarily bad system behavior, even if only one bit is modified. Therefore, safety embedded systems should take measures to prevent and detect such corruptions. Improving the reliability of the root filesystem by reducing writing operations on critical partitions (making a read-only root filesystem) or mirroring the data can be one of preventive measures.

In case of disk corruption detection, the system might be able to automatically repair itself while maintaining file integrity for the critical filesystems (root, kernel filesystems for example).

• Cross-platform development

Embedded devices and development machines can be based on different CPU or system platform architectures. Here are some examples: developing on a Windows x86 machine and deployed on a Linux x86 machine or developing on x86 platform and deployed on ARM based platforms. Handling container development on different platforms can be a challenge, which leads some to develop cross-platform tools.

• System provisioning

As embedded devices do not always have network connectivity or have a restricted connectivity access, the container image may have to be deployed in the firmware image at build time. The provisioning process of the whole system, with the base OS system and the container image, should be accounted for during the product design. In the next section, we will focus on the integration of a container image in the firmware image.

• System update

As the embedded system can evolve with new features, updating a system is also a known challenge in the embedded world. Since the OS system and the container application may not have the same development cycles, it is important to figure out how the update will be done. For example, is it desirable to update the application separately from the base OS ?

• Container traceability and reproductibility

The traceability of Docker containers is an important challenge as the deployment pipeline can sometimes break. Identifying all the container dependencies and modifications, as well as reproducing the build and the deployment histories of each container can save a lot of debugging time. Those concepts are mandatory requirements in the continuous integration paradigm commonly found in the industrial world.

• License compliance

Another challenge is the management of the software licenses. When a container image is distributed, the legal obligations for each file in the image are inherited, including the third-party downloaded files. A full inventory of each layer in every container image has to be generated all the way down to the OS. Complying with license requirements is essential because it minimizes business and legal risk. Yocto building system has already a mechanism to track and maintain compliance with various open source licensing during the lifetime of the product, so the compliance can be done at the OS based system level (see Yocto license compliance). However, it does not exist for an embedded container integrated into Yocto build system.

Conclusion

Embedding a container in a Linux embedded system may have many advantages such as portability, modularity, allowing the separation between the main OS system and the application system and considerably speed up the application development time. But many questions have to be raised during the product design as license compliance, system updating, hardware limitations and cross-platform development.

Savoir-faire Linux has implemented several instances of such a solution with the challenges outlined above.