Resolve crun and runc dependency issues across various distributions

[dtrudg]

Type of issue

Packaging / distros

Description of issue

SingularityCE depends on either runc or crun for OCI-mode functionality. Because we want to offer --no-setgroups functionality, that's only available on crun, we currently prefer crun if both runc and crun are installed.

The versions of crun and runc on distros that we want to aim to support differ. As of 2024-03-12, none of EL 7/8/9, SLES 12/15, Ubuntu 20.04/22.04 provides a working crun. The following complications are known, that we need to try and resolve in a consistent manner:

• runc versions on all of our target distros are suitable to support essential functionality. However, because runc doesn't support our --no-setgroups flag, using runc across the board would cause issues for access host files via supplementary groups, where thes issues can be avoided with crun and --no-setgroups. This is a significant concern given common HPC project file organisation.
• crun <1.14.3 has a bad version check that stops it running bundles with runtime-spec 1.2.0. Due to our dependency chain, we have had to bring in runtime-spec 1.2.0 so this affects us where distros have not yet updated their crun.
• crun >1.11, <1.14.1 has a segfault bug where the optional Linux.Resources structure is not present in a bundle config.json. We currently have a workaround for this in our code.
• crun on Ubuntu 20.04 / 22.04 LTS is quite old, and has issues around directory creation / bind mounts where its behaviour does not match that of runc.
• crun <1.5 is not viable at all, due to the number of behaviour differences vs runc in areas we are e2e-testing etc.
• Docker's own packages for EL8 / EL9 (docker.io) require the containerd.io package. This provides the /usr/bin/runc executable, but does not provide runc, and actually conflicts with distro runc packages. Therefore, our packaging can't Require: runc in this situation.

We either need to think about:

• Bundling a fixed version of crun, that we know works... but this doesn't mesh well with EPEL / Fedora packaging policy.
• A comprehensive solution of version checks in the code, and review of packaging Requires: that avoids problems by choosing crun or runc appropriately for all the above situations.

This is a more difficult problem that it may first appear....