TRAITER LES DÉPENDANCES ANGULAR / NPM SUR LES SYSTÈMES EMBARQUÉS

[L’introduction est en français, le reste du texte en anglais]

Les produits industriels deviennent de plus en plus complexes et nécessitent de beaucoup plus d’interaction et de connectivité. Il est maintenant usuel de faire le suivi et la maintenance d’un appareil à partir d’une interface par une application web embarqué sur ce même appareil. Au cours de la dernière décennie, le développement du web front end a s’est massivement tourné vers les technologies libres émergentes poussées et maintenues par des entités incontournables telles que Google (Angular) ou Facebook (React). Ces technologies permettent de créer facilement des applications web avec des composants d’interface utilisateur complexes avec une connectivité avancée, la gestion des utilisateurs et bien plus encore. Ces frameworks permettent de créer des applications faciles à développer et simples à entretenir tout en optimisant la réutilisation des composants existants. Les développeurs peuvent ainsi obtenir plus de fonctionnalités avec moins de code. Cependant, ils comportent beaucoup de dépendance logicielle et nécessitent un nombre non négligeable de paquets. Actuellement, le tutoriel de base d’Angular a plus de 1000 dépendances et plus de 1200 pour React. Pour gérer ces dépendances, ils utilisent NPM, un gestionnaire de paquets pour les logiciels orientés javascript. De plus, cela s’applique également au backend, car Node.js peut être intégré dans des dispositifs pour servir d’interface entre les applications web et le matériel.

The emergence of package managers

All these frameworks use their own respective package management methods, and thus have their own package lifecycle & the number of packages is growing exponentially, especially when dealing with NPM:

Source: http://www.modulecounts.com/

Each package manager handles all the package dependencies with a fixed version, which means that several versions of a package could be installed on an individual system because package A requires package C version 1.2 while package B requires the same package, but version 1.3.

This dependency tree makes it harder to manage, specifically when dealing with industrial embedded constraints.

Source: https://medium.com/graph-commons/analyzing-the-npm-dependency-network-e2cf318c1d0d

Producing industrial software

Embedded systems require a set of requirements mainly to make them performant, maintainable and resilient to various constraints such as cybersecurity.

An embedded software needs to be customizable

Firmwares can become so complex to assemble that they nowadays require a customizable Linux distribution rather than a generic distribution. These customizations also offer or require a compact firmware size and thus a reduced attack surface for cybersecurity constraints.

An embedded system requires long term reproducible builds.

An industrial product can be maintained over several decades. For this reason the firmware images must be reproducible without relying on external repositories stored on the Internet. Offline builds are definitely a must have, as you can’t rely on source code stored on GitHub or other sites when your product is expected to be rebuilt and deployed over the next 20 years.

Embedded software based on open-source components require these components’ legal requirements to be managed appropriately.

An embedded Linux system typically aggregates a ton of librairies. Any open-source snippet of code is released under the terms of a certain licence. Some of these are permissive (e.g. MIT), while others come with more restrictions (e.g. GPLv3). It is essential for an industrial product distributor to identify all the open-source licenses used in their product, in order to ensure adequate compliance from the legal perspective.

These requirements are well managed and maintained by the Yocto project, a Linux Foundation collaborative open-source project whose goal is to produce tools and processes that enable the creation of Linux distributions, regardless of the hardware architecture.

In essence, the Yocto project offers the ability to use any external source code defining a recipe to package it. A recipe is a .bb file which describes the source code (and its licence) and how to configure, compile and install it.

Here is a simple recipe example:

foobar_git.bb

DESCRIPTION = "FooBar Description"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://COPYING;md5=eb723b61539feef013de476e68b5c50a"

SRC_URI = "git://github.com/example/helloworld.git"

S = "${WORKDIR}/git"

do_patch() {
}

do_compile() {
}

do_install() {
}

The Yocto project splits different steps to ensure a strict separation between the operations that require connectivity, and the operations that do not require any network access. Here are the main operations used:

• do_fetch is an optional operation that fetches the source code from an external repository if the local download directory does not include it
• do_patch offers the ability to customize the source code adding patches
• do_compile compiles your source code for your target
• do_install copies the compilation results on your image

Each recipe also fills in the licence used and its associated  file so that the Yocto project  can inventory the licences used in your product. The Yocto project also manages the dependencies of packages to address more complex software architectures.

The Yocto project provides a tool named devtool to automatically generate a .bb skeleton based on a source code archive. This tool also offers a good integration with software compilation processes such as cmake, autotools and others.

Producing industrial software with NPM

When dealing with an external package manager such as NPM, the constraints required for an industrial image production may not be respected. For example, the npm install command will actually fetch, build and install the dependencies on its own, individually from Yocto’s integrated methods and the benefits they provide. We provided a contribution to the Yocto project to fill this gap.

NPM handles the package definitions and its dependencies with a so-called shrinkwrap file. For instance, here is the definition of a Package A which depends on Package B version 0.0.1  which depends on Package C version 0.1.0:

{
    "name": "A",
    "dependencies": {
        "B": {
            "version": "0.0.1",
            "dependencies": {
                "C": {  "version": "0.1.0" }
            }
        }
    }
}

The Yocto project implemented its own parser for this format with some limitations:

• Only a simpler variant of this shrinkwrap syntax was supported, causing issues when complex packaging semantics were used. For example, Angular’s packaging was failing (we wrote a sample application which reproduces the issues).
  – Scoped packages were failing
  – Package names in uppercase were also generating failures
• The separation of the Yocto project operations was not done properly, as the sources code were fetched by npm during the do_install step. As a consequence:
  – Building without a network connection was not possible.
  – The ability to customize the source code applying patches (do_patch) in the Yocto way was also not possible.
• The aggregation of the Licences was only based on information set in package.json. The standard licence files (Copyright, Licence) were not used.
• The source artifacts were packed by recipe. So a common NPM package used in multiple recipes was fetched multiple times.

Savoir-faire Linux made a full refactoring of NPM management in Yocto. Our 31 patches were merged in January of 2020 and are now available in the Dunfell release of Yocto.

The modifications we brought to the Yocto project included the following changes:

1. The npm fetcher has been splitted to be able to handle the root package and its dependencies differently:
   – A npm:// url will fetch a single package from a registry without its dependencies. It can be used to download a root package just as with git:// or http:// urls.
   – A npmsw:// url will take a shrinkwrap file to fetch all the dependencies.
2. The npm fetchers store the tarballs in a simple way that can be retrieved by other recipes.
3. The do_configure step now creates a local, per-recipe NPM cache which stores the local packages. This local cache is the key to ensure that:
   – The local package sources, potentially previously patched during the do_patch step, are used.
   – No additional network access will be occurring.
4. The do_compile step uses this cache to run the installation process.
5. Every call to npm is heavily controlled to ensure that the correct environment is used and no network access is done out of this controlled context.
6. The usage of devtool has been enhanced to facilitate the integration of external NPM source bundles.
7. The license management bits are now based on Yocto’s guess_license function. All package.json files are also added as extra license files.
8. Tests have been added for both fetchers, recipetool and devtool in order to improve maintainability as the Yocto project evolves.

You can now use devtool to build our sample application using Yocto Dunfell:

source  poky/oe-init-build-env
devtool add "npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest"
devtool build savoirfairelinux-node-server-example

Using this method, we are also able to package an Angular application properly.

We would like to thank the Yocto community for its precious help, and are thrilled to contribute to its rise.