Example recipe(s) with apt removed and RAUC, etc used for updates

[pbrkr]

I've just watched Christopher Obbard's talk from ER 2022 (https://www.youtube.com/watch?v=467kgcSxDf0) and want to look into debos a little more. In the talk, it's recommended that apt is removed for production images and an alternative update system like RAUC, ostree, etc is used.

I think it's important to have a basic example recipe to show how this would work.

[obbardc]

Great idea; we are (slowly) working on adding these kinds of recipes to this repository.

[jluebbe]

When you have an example for RAUC, we'd be happy to link it from our documentation.

[martin-petzold]

I have RAUC integrated and may provide a full example later. Most important for my setup is NOT to brake any Debian system software. IMHO apt can be there and could be used (could also be removed). RAUC then handles A/B rootfs updates. And in my case I have a separate /home partition and another partition ("/state") holding state information. I used bind mounts to link e.g. "/var/lib/iwd" and "/var/lib/bluetooth" to directories on the "/state" partition. The setup is not so complicated, however a lot of potential side effects need to be considered.

[martin-petzold]

Steps to use RAUC with debos:

1. Install RAUC on your build machine (sudo apt-get install rauc)
2. Add RAUC packages to your apt recipe:

• rauc
• rauc-service # this one is required only on the target device

3. Configure bootloader according to: https://rauc.readthedocs.io/en/latest/integration.html

4. Partition your device with two identical partitions for A/B upgrades. Example (the offset is for the booloader):

{{- $architecture := or .architecture "arm64" -}}

{{- $image := or .image "" -}}
{{- $size := or .size "14.0GB" -}}

{{- $offset := or .offset "8MiB" -}}
{{- $rootfs0 := or .rootfs0 "4008MiB" -}}
{{- $rootfs1 := or .rootfs1 "8008MiB" -}}
{{- $state := or .state "8520MiB" -}}
{{- $home := or .home "100%" -}}

architecture: {{ $architecture }}

actions:
  - action: image-partition
    imagename: {{ $image }}
    imagesize: {{ $size }}
    partitiontype: gpt
    mountpoints:
      - mountpoint: /
        partition: rootfs0
      - mountpoint: /state
        partition: state
      - mountpoint: /home
        partition: home
    partitions:
      - name: rootfs0
        fs: ext4
        start: {{ $offset }}
        end: {{ $rootfs0 }}
      - name: rootfs1
        fs: ext4
        start: {{ $rootfs0 }}
        end: {{ $rootfs1 }}
      - name: state
        fs: ext4
        start: {{ $rootfs1 }}
        end: {{ $state }}
      - name: home
        fs: ext4
        start: {{ $state }}
        end: {{ $home }}

5. Add some commands to your build for rootfs image extraction and RAUC bundle creation. Something like this:

• Post action: filesystem-deploy
• All commands need to be postprocess: true
• Only works like this if your default rootfs is the first partition

  ### Create loopback device (/dev/loopX), scan for partitions (/dev/loopXpY), and write path of loopback device to file
  
  - action: run
    postprocess: true
    command: sudo losetup --verbose --find --show --partscan $ARTIFACTDIR/{{ $image }}.img > {{ $loopback }}

  ### Create RAUC upgrade directory

  - action: run
    postprocess: true
    command: mkdir upgrade

  ### Create upgrade image from first partition of loopback device (may be other partition)

  - action: run
    postprocess: true
    command: xargs -i -a {{ $loopback }} find /dev -wholename '{}p*' | sort | head -n 1 | xargs -i sudo dd if={} of=upgrade/{{ $image }}.rootfs.img

  ### Create RAUC manifest file in upgrade directory (see: https://rauc.readthedocs.io/en/latest/examples.html#bundle-generation)

  - action: run
    postprocess: true
    script: script/image/manifest 'upgrade' '{{ $image }}' '{{ $image_id }}' '{{ $image_version }}'

  ### Create RAUC upgrade bundle with upgrade image

  - action: run
    postprocess: true
    command: rauc bundle --cert=/path/to/my.cert.pem --key=/path/to/my.key.pem upgrade $ARTIFACTDIR/{{ $image }}.raucb

  ### Detach loopback device

  - action: run
    postprocess: true
    command: xargs -i -a {{ $loopback }} sudo losetup --verbose --detach {}

  ### Delete file with path of loopback device

  - action: run
    postprocess: true
    command: rm {{ $loopback }}

8. Example for /etc/rauc/system.conf

[system]
compatible=TBD
bootloader=uboot
data-directory=/var/lib/rauc
bundle-formats=verity

[keyring]
path=/path/to/my.cert.pem

[slot.rootfs.0]
device=/dev/mmcblk2p1
type=ext4
bootname=A

[slot.rootfs.1]
device=/dev/mmcblk2p2
type=ext4
bootname=B

9. I suggest to use bind mounts (I am using systemd *.mount files) to move state to be maintained across rootfs upgrades to a separate partition. This is especially important for the RAUC state (location defined in system.conf), in this case /var/lib/rauc. Example for /usr/lib/systemd/system/var-lib-rauc.mount (be aware that /state is a separate partition):

[Unit]
Description=Mount RAUC state

[Mount]
What=/state/rauc
Where=/var/lib/rauc
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

In case you need a more recent RAUC, you can try to install it from bookworm repository or (of course) build from source.

I will maybe suggest some documentation and maybe a dedicated action for upgrade management later.

[martin-petzold]

Still one issue seem to be the UUIDs in the /etc/fstab. If only rootfs is upgraded, then the UUIDs in the /etc/fstab from the new image don't match the partitions on the system. I am trying to find out, how debos could write the device names instead of the UUIDs. However, this maybe only required for the other partitions (home and state) because for the rootfs partition it should be correct.

[andhe]

FWIW I've done similar setups using Mender (by northern.tech) a bunch of times if anyone is interested feel free to ask me anything.

Quickly glancing over the above RAUC setup a pitfall I've hit is when parted rounds partition sizes to different amount of sectors and rootfs0 and rootfs1 thus does not become equal size (in sectors, even though they are specified as same size in MiB, which can explode when deploying an image and rootfs1 is smaller than rootfs0), so that might be something to watch out for.

[martin-petzold]

btw. I had systemd issued with bind mounts in separate *.mount files. I then used definition of bind mounts in the /etc/fstab.