From ecc214f2d440373776477ac98a0befdc3b04ab0d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Emmanuel=20Fr=C3=A9con?= Date: Thu, 17 Mar 2022 08:55:19 +0100 Subject: [PATCH] Improve documentation --- Dockerfile | 6 ++- README.md | 113 ++++++++++++++++++++++++++++++++++++++-------- systemd/README.md | 23 ++++++++++ 3 files changed, 123 insertions(+), 19 deletions(-) create mode 100644 systemd/README.md diff --git a/Dockerfile b/Dockerfile index b9e1f91..993e881 100644 --- a/Dockerfile +++ b/Dockerfile @@ -9,6 +9,9 @@ ARG GH_RUNNER_VERSION="latest" ARG COMPOSE_VERSION=latest ARG COMPOSE_SWITCH_VERSION=latest +# Copy our "library" and all necessary shell-wrappers. Note that some of the +# wrappers that we install will be used to build the image itself, i.e. as part +# of some of the `RUN` commands below. COPY lib/*.sh /usr/local/share/runner/ COPY *.sh /usr/local/bin/ @@ -110,5 +113,6 @@ RUN for s in systemd/runner*.service; do \ /etc/systemd/system/multi-user.target.wants/; \ done -# Set systemd as entrypoint. +# Set systemd as entrypoint. There will hardly be any logs, use `journalctl` +# from within instead. ENTRYPOINT [ "/sbin/init", "--log-level=err" ] diff --git a/README.md b/README.md index 07c1f62..1771a93 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,13 @@ # GitHub Actions Runner -This project implements a self-hosted GitHub Actions Runner. The project -implements a Docker image for setting up, running and [registering][register] a -runner within an Organisation or a Repository. This image **depends** on -[sysbox], an alternative OCI runtime. [sysbox] makes it possible to run Docker -in Docker (DinD) without having to rely on elevated privileges. - +This project implements a self-hosted GitHub Actions Runner, tuned for use from +within Kubernetes clusters. The project implements an [Ubuntu][ubuntu]-based +Docker image for setting up, running and [registering][register] a runner within +an Organisation or a Repository. This image **requires** [sysbox], an +alternative OCI runtime. [sysbox] makes it possible to run Docker in Docker +(DinD) without having to rely on elevated privileges. + + [ubuntu]: https://hub.docker.com/_/ubuntu [register]: https://docs.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners [sysbox]: https://github.com/nestybox/sysbox @@ -14,20 +16,24 @@ in Docker (DinD) without having to rely on elevated privileges. * Dynamic registration of the runner at GitHub. * Runner registration for an entire organisation, or for a repository. * Support for both enterprise installations, and `github.com`. -* Support for labels and groups to categorise runners. +* Support for [labels] and [groups] to categorise runners. * Able to run all [types] of actions, including [Docker][container-action] container actions! -* Multi-platform support. -* Each runner can be customised through running a series of script/programs - prior to registration at the GitHub server. +* Multi-platform [support](#supported-arhitectures). +* Each runner can be customised/configured through running a series of + script/programs prior to registration at the GitHub server. * Automatically [follows](#releases) the [release] tempo of the official [runner]. Generated images will be tagged with the SemVer of the release. -* `latest` tag will correspond to latest [release] of the [runner]. +* `latest` tag will correspond to latest [release] of the [runner] and this + project. * Fully automated [workflows](.github/workflows/README.md), manual interaction possible. * Comes bundled with latest `docker compose` (v2, the plugin), together with the - `docker-compose` [shim]. + `docker-compose` [shim] and a few other essential [tools](#available-tools). +* Within containers, workflows are executed by a regular user called `runner`. + [labels]: https://docs.github.com/en/actions/hosting-your-own-runners/using-labels-with-self-hosted-runners + [groups]: https://docs.github.com/en/actions/hosting-your-own-runners/managing-access-to-self-hosted-runners-using-groups [types]: https://docs.github.com/en/actions/creating-actions/about-custom-actions#types-of-actions [container-action]: https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action [release]: https://github.com/actions/runner/releases @@ -36,6 +42,9 @@ in Docker (DinD) without having to rely on elevated privileges. ## Environment Variables +This runner recognises the following environment variables for its +configuration. + | Environment Variable | Description | | --- | --- | | `RUNNER_NAME` | The name of the runner to use. Supercedes (overrides) `RUNNER_NAME_PREFIX` | @@ -55,9 +64,10 @@ in Docker (DinD) without having to rely on elevated privileges. ## Available Tools -These images do **not** contain **all** the tools that GitHub offers by default -in their runners. Workflows might work improperly when running from within these -runners. The [Dockerfile](./Dockerfile) for the runner images ensures: +This images does **not** contain **all** the tools that GitHub offers by default +in their runners. Workflows might work improperly when running from within +runners created using this image. The [Dockerfile](./Dockerfile) for the runner +images ensures: * An installation of the Docker daemon, including the `docker` cli binary. * An installation of Docker [compose]. Unless otherwise specified, the latest @@ -65,11 +75,78 @@ runners. The [Dockerfile](./Dockerfile) for the runner images ensures: At the time of writing, this installs the latest `2.x` branch, rewritten in golang, including the `docker-compose` compatibility [shim]. * An installation of `git` that is compatible with the github runner code. - Unless otherwise specified, the latest stable version at the time of image - building will be automatically picked up. This is because the default version - available in Ubuntu is too old. + Unless otherwise specified, the latest stable [version][git-release] at the + time of image building will be automatically picked up. This is because the + default version available in Ubuntu is too old. * The `build-essential` package, in order to facilitate compilation. + [compose]: https://github.com/docker/compose + [git-release]: https://launchpad.net/~git-core/+archive/ubuntu/ppa + +## Releases + +By default,the image follows the release tempo of the main [runner][release] +project. New images with the semantic version as a tag will be made available +shortly after a new [release] is made. There also exist tags that combine the +version of the runner and the short commit SHA of this project. These tags are +aimed at picking the proper combination, if necessary and when changes have been +applied to this project, which happens seldom. See +[here](.github/workflows/README.md) for more details. + +## Known Limitations + +### Logging + +Containers created with this image will not have any log output. This is because +the runner (and the services that are started prior to the runner) are started +as systemd units. To get their logs, you will have to jump into the container, +and from an interactive shell (`bash` available) request for their logs through +`journalctl`. See [this](./systemd/README.md) for the list of relevant services. + +### PV Mounts + +When running from within kubernetes, you might not be able to use persistent +volumes, e.g. for the tool cache or the working directory. After some time, you +might loose the mount inside the container. This has been reported when `sysbox` +was installed on older kernels (Ubuntu 20.04 LTS) and with `shiftfs`. Using +local mounts, e.g. [`emptyDir`][emptyDir] works. + + [emptyDir]: https://kubernetes.io/docs/concepts/storage/volumes/#emptydir + +### Coexistence of `sysbox` Containers + +Our experience has shown that it is not always possible to run several `sysbox` +containers on the same Kubernetes node. To circumvent this limitation, you +should: + +* Arrange for kubernetes to terminate Pods before is starts new ones (otherwise, + there might be two Pods running on the same node during rollouts: one waiting + to be up, and one still there, soon to be terminated). +* Force spreading your Pods across the nodes of your cluster. + +To enforce termination, add the following snippet to your Deployment +specification: + +```yaml + strategy: + type: Recreate +``` + +To enforce spreading across nodes, add the following snippet to your Pod +specification. This creates topology groups of one single member, per the name +of the node. You should adapt the label to whatever label you have chosen. + +```yaml + topologySpreadConstraints: + - + maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + app: gh-runner-sysbox +``` + ## Acknowledgments This image takes a lot of inspiration from: diff --git a/systemd/README.md b/systemd/README.md new file mode 100644 index 0000000..9bcfa1d --- /dev/null +++ b/systemd/README.md @@ -0,0 +1,23 @@ +# `systemd` units + +This directory contains the variour `systemd` services that are installed and +enabled at the runner. These are (in order of execution): + ++ `runner-conf` will steal most of the environment of the main process, i.e. + `systemd` itself and propagate this environment to a file, so that it can be + picked up by other services below. This complexity is necessary in order to + support the configuration of the runner from environment variables. When + `systemd` units are started, they always start from a clean environment, so we + need a mechanism to capture the environment of the main process, which is set + at startup by the container runtime. Implementation is through + [`export.sh`](../export.sh). ++ `runner-preflight` inherits the environment from above and runs executables + pointed at by the `RUNNER_PREFLIGHT_PATH` variable. These will be run by the + `runner` user, meaning that you will have to `sudo` if you want to perform + "system" work. As this runs before the Docker daemon is started, it is + possible to amend/install Docker settings at this point, but also to install + other services, libraries, packages that would be necessary for the runner. + Implementation is through [`hook.sh`](../hook.sh). ++ `runner` is the main service and in charge of starting the runner. The service + will pick the environment from above (first service) and start the runner, via + [`runner.sh`](../runner.sh), as the `runner` user.