Concept: Separate Application / Container / Artifact Updates in RAUC

[ejoerns]

In the past, there have been multiple requests and use cases for updating the application or data files (referred to as 'artifacts' in the future) without the need to update the rootfs.
So, far there is no ready-to-use concept for this since RAUC bundles are designed to define the entire target state of the system (and handled as such).

Since the rootfs should still be defined by the image contained in the bundle, RAUC should not simple write the artifacts into the rootfs.
Artifacts will be written into a separate slot (partition) only. For an alternative to modifying files in the rootfs, see further down.

From our perspective, there are two types of artifacts that must be handled differently:

• artifacts without dependencies into the rootfs: typically container, or data files (e.g. navigation maps)
• artifacts with dependencies into the rootfs: typically applications built against system libraries

For the first case (no dependency) there can be a single artifact partition for all rootfs partitions.
This makes handling (like mounting) easy.

For the second case (with dependencies) there should be an artifact partition for each rootfs partition.
So that when falling back, application and libraries still match.

A partition should hold one or many artifacts that each can be updated individually and, as other updates done with RAUC, artifact updates also need to be atomic.

To achieve this, RAUC must not create or format the file system of the artifact partition. For updating the artifacts inside this file systems, different techniques can apply, depending on the type of artifact (data):

• single files: can be updated atomically with copy&rename
• directory trees: can be updated atomically with ostree or copy&rename
• docker container: can be updated with 'docker import' from the bundle artifact

For the above-mentioned use case of overriding files in the rootfs, two possible solutions exist:

• bind mount the file to the file in the artifact partition
• use directory from artifact partition as seed for an overlayfs

The syntax could look as follows:

system.conf:

[slot.rootfs.0]
device=/dev/sda1
type=ext4

[slot.rootfs.1]
device=/dev/sda2
type=ext4

[slot.myartifacts]
device=/dev/sda3
type=artifacts

manifest:

[image.myartifacts]
filename=app1.docker
filename=app2.run

An alternative could be to define a slot for each artifact. Naming this artifacts is just an initial proposal.

This is just a draft concept. Comments and opinions are welcome.

[jluebbe]

We intend to work on a first part of this for container updates in a shared partition.

[JSydll]

First of all: thanks for your efforts! rauc has already evolved to a quite impressive framework.

Concerning the concept, I had a few questions coming up in my mind:

What are the major drivers for the concept?

If it is allowed to only include artifacts and no rootfs in an update, this would allow the rootfs and the artifacts to diverge over time and I can imagine all kinds of issues arising from that. How to ensure consistency over time? How to verify the bundle versus all available system configurations in the field?

What is the distinction to single-artifact-focused approaches like package managers?

In my experience, dependencies do not only occur between applications and rootfs libs, but also between different applications. And there's the time dimension, too...

[jluebbe]

First of all: thanks for your efforts! rauc has already evolved to a quite impressive framework.

Thanks! :)

Concerning the concept, I had a few questions coming up in my mind:

What are the major drivers for the concept?

If it is allowed to only include artifacts and no rootfs in an update, this would allow the rootfs and the artifacts to diverge over time and I can imagine all kinds of issues arising from that. How to ensure consistency over time? How to verify the bundle versus all available system configurations in the field?

We already allow do update non-rootfs slots from bundles which include no rootfs image, but recommend against that. A bundle should always describe the full intended state of the target system and the same applies to artifact updates.

So you'd still include a rootfs image in a bundle with artifacts, but it would only be installed when it differs from the already installed version. With streaming, that means that the rootfs is also only downloaded when needed.

If you have some other component which tracks versions in the field, that component could consider more relaxed compatibility relationships and generate device-specific bundles (with a compat check in a hook perhaps). That's out of scope for RAUC itself, though.

(Some time ago we thought about adding slot content checks to RAUC itself, so it would be able to check the image hash even if the image itself was not part of the bundle. This "partial bundle" would be smaller but would not allow unintended combinations in the field. Since we have streaming now, it's not as necessary anymore, though.)

What is the distinction to single-artifact-focused approaches like package managers?

In my experience, dependencies do not only occur between applications and rootfs libs, but also between different applications. And there's the time dimension, too...

You'd "resolve" the dependencies when creating the bundle, so you'd end up with a fixed set of rootfs+artifacts to be installed.

Compared to a package manager, RAUC installs the artifacts atomically. Additionally, even when bound to the A/B rootfs slot, there's still only one copy per artifact version stored, but linked from the rootfs parent slot as needed. This is useful for cases where you have limited space. You'd use a small rootfs (containing RAUC and some management infrastructure), but use a shared partition for artifacts and save space through deduplication.

If you have enough space to store everything into A/B rootfs slots and you're building everything from the same buildsystem anyway, not using artifacts would be simpler. :)