Dealing with Angular / NPM dependencies on embedded systems

Industrial products have become immensely more complex and require much more interaction and connectivity. It is now very common to control a device from a dashboard provided by an internal web application.

Over the last decade, frontend web development drastically migrated to emerging open-source frameworks pushed and maintained by big players such as Google (Angular) or Facebook (React). Allowing to easily create web applications with complex UI components, advanced connectivity, user management and much more.

These frameworks provide the capability to create Single Page Application in a very clean and maintainable way and to allow component reusability. The developers can then achieve more functionality with less code.

But, they both depend and or rely on a lot of packages. Currently, the basic tutorial from Angular has more than 1000 dependencies and more than 1200 for React. To manage these dependencies they utilize NPM which is a package manager for javascript oriented software.

Moreover, this also applies to the backend as Node.js can be embedded in devices to act as the link between the web applications and the hardware.

The emergence of package managers

All these frameworks use their own respective package management methods, and thus have their own package lifecycle & the number of packages is growing exponentially, especially when dealing with NPM:

Source: http://www.modulecounts.com/

Each package manager handles all the package dependencies with a fixed version, which means that several versions of a package could be installed on an individual system because package A requires package C version 1.2 while package B requires the same package, but version 1.3.

This dependency tree makes it harder to manage, specifically when dealing with industrial embedded constraints.

Source: https://medium.com/graph-commons/analyzing-the-npm-dependency-network-e2cf318c1d0d

Producing industrial software

Embedded systems require a set of requirements mainly to make them performant, maintainable and resilient to various constraints such as cybersecurity.

An embedded software needs to be customizable

Firmwares can become so complex to assemble that they nowadays require a customizable Linux distribution rather than a generic distribution. These customizations also offer or require a compact firmware size and thus a reduced attack surface for cybersecurity constraints.

An embedded system requires long term reproducible builds.

An industrial product can be maintained over several decades. For this reason the firmware images must be reproducible without relying on external repositories stored on the Internet. Offline builds are definitely a must have, as you can’t rely on source code stored on GitHub or other sites when your product is expected to be rebuilt and deployed over the next 20 years.

Embedded software based on open-source components require these components’ legal requirements to be managed appropriately.

An embedded Linux system typically aggregates a ton of librairies. Any open-source snippet of code is released under the terms of a certain licence. Some of these are permissive (e.g. MIT), while others come with more restrictions (e.g. GPLv3). It is essential for an industrial product distributor to identify all the open-source licenses used in their product, in order to ensure adequate compliance from the legal perspective.

These requirements are well managed and maintained by the Yocto project, a Linux Foundation collaborative open-source project whose goal is to produce tools and processes that enable the creation of Linux distributions, regardless of the hardware architecture.

In essence, the Yocto project offers the ability to use any external source code defining a recipe to package it. A recipe is a .bb file which describes the source code (and its licence) and how to configure, compile and install it.

Here is a simple recipe example:

foobar_git.bb

DESCRIPTION = "FooBar Description"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://COPYING;md5=eb723b61539feef013de476e68b5c50a"

SRC_URI = "git://github.com/example/helloworld.git"

S = "${WORKDIR}/git"

do_patch() {
}

do_compile() {
}

do_install() {
}

The Yocto project splits different steps to ensure a strict separation between the operations that require connectivity, and the operations that do not require any network access. Here are the main operations used:

• do_fetch is an optional operation that fetches the source code from an external repository if the local download directory does not include it
• do_patch offers the ability to customize the source code adding patches
• do_compile compiles your source code for your target
• do_install copies the compilation results on your image

Each recipe also fills in the licence used and its associated  file so that the Yocto project  can inventory the licences used in your product. The Yocto project also manages the dependencies of packages to address more complex software architectures.

The Yocto project provides a tool named devtool to automatically generate a .bb skeleton based on a source code archive. This tool also offers a good integration with software compilation processes such as cmake, autotools and others.

Producing industrial software with NPM

When dealing with an external package manager such as NPM, the constraints required for an industrial image production may not be respected. For example, the npm install command will actually fetch, build and install the dependencies on its own, individually from Yocto’s integrated methods and the benefits they provide. We provided a contribution to the Yocto project to fill this gap.

NPM handles the package definitions and its dependencies with a so-called shrinkwrap file. For instance, here is the definition of a Package A which depends on Package B version 0.0.1  which depends on Package C version 0.1.0:

{
    "name": "A",
    "dependencies": {
        "B": {
            "version": "0.0.1",
            "dependencies": {
                "C": {  "version": "0.1.0" }
            }
        }
    }
}

The Yocto project implemented its own parser for this format with some limitations:

• Only a simpler variant of this shrinkwrap syntax was supported, causing issues when complex packaging semantics were used. For example, Angular’s packaging was failing (we wrote a sample application which reproduces the issues).
  – Scoped packages were failing
  – Package names in uppercase were also generating failures
• The separation of the Yocto project operations was not done properly, as the sources code were fetched by npm during the do_install step. As a consequence:
  – Building without a network connection was not possible.
  – The ability to customize the source code applying patches (do_patch) in the Yocto way was also not possible.
• The aggregation of the Licences was only based on information set in package.json. The standard licence files (Copyright, Licence) were not used.
• The source artifacts were packed by recipe. So a common NPM package used in multiple recipes was fetched multiple times.

Savoir-faire Linux made a full refactoring of NPM management in Yocto. Our 31 patches were merged in January of 2020 and are now available in the Dunfell release of Yocto.

The modifications we brought to the Yocto project included the following changes:

1. The npm fetcher has been splitted to be able to handle the root package and its dependencies differently:
   – A npm:// url will fetch a single package from a registry without its dependencies. It can be used to download a root package just as with git:// or http:// urls.
   – A npmsw:// url will take a shrinkwrap file to fetch all the dependencies.
2. The npm fetchers store the tarballs in a simple way that can be retrieved by other recipes.
3. The do_configure step now creates a local, per-recipe NPM cache which stores the local packages. This local cache is the key to ensure that:
   – The local package sources, potentially previously patched during the do_patch step, are used.
   – No additional network access will be occurring.
4. The do_compile step uses this cache to run the installation process.
5. Every call to npm is heavily controlled to ensure that the correct environment is used and no network access is done out of this controlled context.
6. The usage of devtool has been enhanced to facilitate the integration of external NPM source bundles.
7. The license management bits are now based on Yocto’s guess_license function. All package.json files are also added as extra license files.
8. Tests have been added for both fetchers, recipetool and devtool in order to improve maintainability as the Yocto project evolves.

You can now use devtool to build our sample application using Yocto Dunfell:

source  poky/oe-init-build-env
devtool add "npm://registry.npmjs.org;package=@savoirfairelinux/node-server-example;version=latest"
devtool build savoirfairelinux-node-server-example

Using this method, we are also able to package an Angular application properly.

We would like to thank the Yocto community for its precious help, and are thrilled to contribute to its rise.