From e260d42304a5822816ebc1752ab6c0818b9ee589 Mon Sep 17 00:00:00 2001 From: Dmitry Tantsur Date: Mon, 8 Apr 2024 20:25:21 +0200 Subject: [PATCH] Live updates (servicing) for provisioned hosts This design proposal introduces the Ironic servicing feature into BMO. Servicing is a process of modifying already provisioned nodes, e.g. changing BIOS settings or updating firmware (the 2 features I want to concentrate on right now). Signed-off-by: Dmitry Tantsur --- .../baremetal-operator/host-live-updates.md | 194 ++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 design/baremetal-operator/host-live-updates.md diff --git a/design/baremetal-operator/host-live-updates.md b/design/baremetal-operator/host-live-updates.md new file mode 100644 index 00000000..b9e77840 --- /dev/null +++ b/design/baremetal-operator/host-live-updates.md @@ -0,0 +1,194 @@ + + +# Live updates of hosts + +## Status + +One of: implementable + +## Summary + +This proposal enables baremetal-operator to conduct certain actions on already +provisioned BareMetalHosts. + +## Motivation + +Currently, all preparation actions, such as BIOS settings, hardware RAID +configuration and newly firmware updates, happen in the `preparing` state +before a host is provisioned. While it makes perfect sense for RAID, firmware +updates can normally happen for already provisioned hosts. + +We already have a precedent of modifying running hosts: a `DataImage` can +be attached on reboot. This proposal extends the same to other actions, +including potential future ones. This new feature will be called *live updates* +throughout this document. + +### Goals + +- Firmware updates and BIOS settings changes are possible for already + provisioned hosts. +- Operators can opt out of live updates using RBAC. + +### Non-Goals + +- Changing hardware RAID configuration. +- Covering data images in the same proposal (we may add them later). +- Supporting actions from inside the running operating system. +- Live updates without rebooting. + +## Proposal + +A new auxiliary CRD `HostUpdatePolicy` will define policies for different +actions (initially, firmware updates and firmware settings). +The defaults will match the current ones. A policy object will be matched to +a BMH by its name (similarly to other resources). RBAC can be used to prevent +a certain user from creating/changing policies. + +### User Stories + +#### Story 1 + +As a BareMetalHost user, I want to be able to update system firmware or change +firmware settings without de-provisioning my hosts. + +#### Story 2 + +As a cluster operator, I want to control whether certain users can run live +updates on BareMetalHosts. + +## Design Details + +### Custom Resource Definitions + +A new auxiliary CRD `HostUpdatePolicy` will be added with the following Go +definition (shortened): + +```go +type HostUpdatePolicySpec struct { + FirmwareSettings HostUpdatePolicyType + FirmwareUpdates HostUpdatePolicyType +} +``` + +`HostUpdatePolicyType` is an enumeration with the following values: + +- `onPreparing` (the default) - updates will only happen in the `preparing` + state (no live updates). +- `onReboot` - updates will happen on the next reboot. + +In all cases, detached hosts will need to be attached first. Hosts not in the +`provisioned` state will not be affected. + +### New Operational Status + +A new operational status `servicing` will be added to designate hosts that are +undergoing a live update. Once the update finishes, the operational status will +become `OK` or `error`. + +### Example workflow + +As an operator, I want to update firmware on my Kubernetes worker node. My +actions: + +1. Mark the affected node as non-schedulable, drain the workloads from it. +2. Find the corresponding `HostFirmwareComponents` resource, edit it to put + an HTTP link to the expected firmware image(s). +3. Edit the `BareMetalHost` resource to add the reboot annotations. +4. Wait for the host's `operationalStatus` field to first change to + `servicing`, then back to `OK`. +5. Wait for the node's status to become `Ready` again. +6. Mark the node as schedulable. + +### Implementation Details/Notes/Constraints + +- The power management code for provisioned hosts will be updated with a logic + to check servicing on each power on. If the policy allows servicing, and + there is a change in the corresponding resources (`HostFirmwareSetting` etc), + the [servicing](https://docs.openstack.org/ironic/latest/admin/servicing) + operation will be invoked. + + Servicing is technically very similar to cleaning, but runs on already + provisioned hosts. It reboots into the IPA ramdisk (if needed) and runs + a sequence of requested steps. As with cleaning, BMO will be responsible for + creating a list of steps to execute. Unlike cleaning, the end state of the + host is provisioned and powered on. + + In case of firmware updates, Ironic downloads the requested images into the + local HTTP server and provides links to them to the machine via Redfish. Then + the machine is rebooted. + +- Although the BMO's reboot annotation will be used as a trigger, + the actual reboot will not be handled by BMO in this case. All reboots + (at least two in most cases) will be orchestrated by Ironic in reality. + BMO will simply remove the reboot annotation after triggering servicing. + +- Draining hosts will not be handled by BMO. It will be entirely up to the user + to ensure that the hosts can be updated. + + In case of a Kubernetes cluster, the workloads will need to be drained and + the node cordoned before doing the reboot. After the host returns to the + operational status `OK`, the node will need to be brought back into the + cluster. + +- The policy resources will have neither a status nor a controller. + +### Risks and Mitigations + +- Adding new operational status is potentially breaking for consumers that + expect to always see `OK`. Since BMH API is still v1alpha1, it's probably + acceptable. On the other hand, adding a new state would be too risky. + +### Work Items + +- Add new CRD to the API. +- Update gophercloud to a version supporting servicing. +- Implement a `Service` call in the provisioners (similarly to `Prepare`). +- Modify the power management code to enable servicing. +- Write an end-to-end test of changing firmware settings using the redfish + emulator. + +### Dependencies + +- Modifying provisioned hosts relies on the Ironic servicing feature + introduced in API version 1.87. It functions very similarly to cleaning + but acts on nodes in the ``active`` state. + +- GopherCloud support for servicing will be introduced in the next version + after 2.0-beta3. + +- Ironic will need to be extended to support firmware updates during servicing. + +### Test Plan + +- The redfish emulator supports changing BIOS settings. Thus, we can write + an e2e test that exercises this feature. + +### Upgrade / Downgrade Strategy + +- The feature is completely opt-in. + +### Version Skew Strategy + +- Ironic API version 1.87 will be required for live updates to function. Live + updates will be ignored without it. + +## Drawbacks + +- The implementation will complicate the BMO code quite significantly since + we'll have to check the Ironic's state machine for already provisioned hosts + and react to errors. + +## Alternatives + +- Recommend operators to use in-band firmware update / settings tooling. It may + or may not exist for given hardware. + +## References + +[Proof of concept (without firmware +updates)](https://github.com/metal3-io/baremetal-operator/pull/1689)