Support patching installer manifests

With this change, you can use new API field
`ClusterDeployment.Spec.Provisioning.CustomizationRef` to point to a
ClusterDeploymentCustomization (hereinafter "CDC") object in the same
namespace as the ClusterDeployment (CD).

CDC accepts a new subfield, `Spec.InstallerManifestPatches`, which
consists of:
- `Glob`: a string representing a file glob, relative to the installer
  working directory, matching one or more manifest files.
- `Patches`: a list of `PatchEntity` representing RFC6902 JSON patches
  to apply to the matched manifest(s).

ClusterPools:
CDC was already being used by ClusterPool-owned CDs to allow patching
the install-config generated from the template referred to by
`ClusterPool.Spec.InstallConfigSecretTemplateRef`. With this change,
ClusterPool-owned CDs can start using manifest patches in two ways (not
mutually exclusive):
- Patches specific to the CD can be included in the
  `InstallerManifestPatches` field of the existing Inventory CDCs.
- Patches applicable to all CDs in the pool can be provided by a CDC
  referenced via a new ClusterPool.Spec.CustomizationRef field.

HIVE-1793

apis/hive/v1/clusterdeployment_types.go

@@ -218,6 +218,12 @@ type Provisioning struct {
 	// +optional
 	InstallConfigSecretRef *corev1.LocalObjectReference `json:"installConfigSecretRef,omitempty"`
 
+	// CustomizationRef is a reference to a ClusterDeploymentCustomization containing
+	// InstallerManifestPatches to be applied to the manifests generated by openshift-install prior
+	// to starting the installation. (InstallConfigPatches will be ignored -- those changes should
+	// be made directly to the install-config.yaml referenced by InstallConfigSecretRef.)
+	CustomizationRef *corev1.LocalObjectReference `json:"customizationRef,omitempty"`
+
 	// ReleaseImage is the image containing metadata for all components that run in the cluster, and
 	// is the primary and best way to specify what specific version of OpenShift you wish to install.
 	ReleaseImage string `json:"releaseImage,omitempty"`

apis/hive/v1/clusterdeploymentcustomization_types.go

@@ -43,6 +43,27 @@ type ClusterDeploymentCustomization struct {
 type ClusterDeploymentCustomizationSpec struct {
 	// InstallConfigPatches is a list of patches to be applied to the install-config.
 	InstallConfigPatches []PatchEntity `json:"installConfigPatches,omitempty"`
+
+	// InstallerManifestPatches is a list of patches to be applied to installer-generated manifests.
+	InstallerManifestPatches []InstallerManifestPatch `json:"installerManifestPatches,omitempty"`
+}
+
+type InstallerManifestPatch struct {
+	// ManifestSelector identifies one or more manifests to patch
+	ManifestSelector ManifestSelector `json:"manifestSelector"`
+
+	// Patches is a list of RFC6902 patches to apply to manifests identified by manifestSelector.
+	Patches []PatchEntity `json:"patches"`
+}
+
+type ManifestSelector struct {
+	// Glob is a file glob (per https://pkg.go.dev/path/filepath#Glob) identifying one or more
+	// manifests. Paths should be relative to the installer's working directory. Examples:
+	// - openshift/99_role-cloud-creds-secret-reader.yaml
+	// - openshift/99_openshift-cluster-api_worker-machineset-*.yaml
+	// - */*secret*
+	// It is an error if a glob matches zero manifests.
+	Glob string `json:"glob"`
 }
 
 // PatchEntity represents a json patch (RFC 6902) to be applied

apis/hive/v1/clusterpool_types.go

@@ -102,6 +102,12 @@ type ClusterPoolSpec struct {
 	// additional features of the installer.
 	// +optional
 	InstallerEnv []corev1.EnvVar `json:"installerEnv,omitempty"`
+
+	// CustomizationRef refers to a ClusterDeploymentCustomization object whose InstallerManifestPatches should
+	// be applied to *all* ClusterDeployments created by this ClusterPool. This is in addition to any CDC from
+	// Inventory. The CDC must exist in the ClusterPool's namespace. It will be copied to the namespace of each
+	// ClusterDeployment generated by the ClusterPool.
+	CustomizationRef *corev1.LocalObjectReference `json:"customizationRef,omitempty"`
 }
 
 type HibernationConfig struct {

config/crds/hive.openshift.io_clusterdeploymentcustomizations.yaml

@@ -80,6 +80,73 @@ spec:
                   - path
                   type: object
                 type: array
+              installerManifestPatches:
+                description: InstallerManifestPatches is a list of patches to be applied
+                  to installer-generated manifests.
+                items:
+                  properties:
+                    manifestSelector:
+                      description: ManifestSelector identifies one or more manifests
+                        to patch
+                      properties:
+                        glob:
+                          description: |-
+                            Glob is a file glob (per https://pkg.go.dev/path/filepath#Glob) identifying one or more
+                            manifests. Paths should be relative to the installer's working directory. Examples:
+                            - openshift/99_role-cloud-creds-secret-reader.yaml
+                            - openshift/99_openshift-cluster-api_worker-machineset-*.yaml
+                            - */*secret*
+                            It is an error if a glob matches zero manifests.
+                          type: string
+                      required:
+                      - glob
+                      type: object
+                    patches:
+                      description: Patches is a list of RFC6902 patches to apply to
+                        manifests identified by manifestSelector.
+                      items:
+                        description: PatchEntity represents a json patch (RFC 6902)
+                          to be applied
+                        properties:
+                          from:
+                            description: From is the json path to copy or move the
+                              value from
+                            type: string
+                          op:
+                            description: Op is the operation to perform.
+                            enum:
+                            - add
+                            - remove
+                            - replace
+                            - move
+                            - copy
+                            - test
+                            type: string
+                          path:
+                            description: Path is the json path to the value to be
+                              modified
+                            type: string
+                          value:
+                            description: |-
+                              Value is the *string* value to be used in the operation. For more complex values, use
+                              ValueJSON.
+                            type: string
+                          valueJSON:
+                            description: |-
+                              ValueJSON is a string representing a JSON object to be used in the operation. As such,
+                              internal quotes must be escaped. If nonempty, Value is ignored.
+                            format: byte
+                            type: string
+                        required:
+                        - op
+                        - path
+                        type: object
+                      type: array
+                  required:
+                  - manifestSelector
+                  - patches
+                  type: object
+                type: array
             type: object
           status:
             description: ClusterDeploymentCustomizationStatus defines the observed

config/crds/hive.openshift.io_clusterdeployments.yaml

@@ -1009,6 +1009,24 @@ spec:
                   Provisioning contains settings used only for initial cluster provisioning.
                   May be unset in the case of adopted clusters.
                 properties:
+                  customizationRef:
+                    description: |-
+                      CustomizationRef is a reference to a ClusterDeploymentCustomization containing
+                      InstallerManifestPatches to be applied to the manifests generated by openshift-install prior
+                      to starting the installation. (InstallConfigPatches will be ignored -- those changes should
+                      be made directly to the install-config.yaml referenced by InstallConfigSecretRef.)
+                    properties:
+                      name:
+                        default: ""
+                        description: |-
+                          Name of the referent.
+                          This field is effectively required, but due to backwards compatibility is
+                          allowed to be empty. Instances of this type with an empty value here are
+                          almost certainly wrong.
+                          More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+                        type: string
+                    type: object
+                    x-kubernetes-map-type: atomic
                   imageSetRef:
                     description: |-
                       ImageSetRef is a reference to a ClusterImageSet. If a value is specified for ReleaseImage,

config/crds/hive.openshift.io_clusterpools.yaml

@@ -96,6 +96,24 @@ spec:
                     pattern: ^([0-9]+(\.[0-9]+)?(ns|us|µs|ms|s|m|h))+$
                     type: string
                 type: object
+              customizationRef:
+                description: |-
+                  CustomizationRef refers to a ClusterDeploymentCustomization object whose InstallerManifestPatches should
+                  be applied to *all* ClusterDeployments created by this ClusterPool. This is in addition to any CDC from
+                  Inventory. The CDC must exist in the ClusterPool's namespace. It will be copied to the namespace of each
+                  ClusterDeployment generated by the ClusterPool.
+                properties:
+                  name:
+                    default: ""
+                    description: |-
+                      Name of the referent.
+                      This field is effectively required, but due to backwards compatibility is
+                      allowed to be empty. Instances of this type with an empty value here are
+                      almost certainly wrong.
+                      More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+                    type: string
+                type: object
+                x-kubernetes-map-type: atomic
               hibernateAfter:
                 description: |-
                   HibernateAfter will be applied to new ClusterDeployments created for the pool. HibernateAfter will transition

docs/enhancements/clusterpool-inventory.md

@@ -2,28 +2,27 @@
 
 [HIVE-1367](https://issues.redhat.com/browse/HIVE-1367)
 
-- [Clusterpool for on-prem cloud providers](#clusterpool-for-on-prem-cloud-providers)
-  - [Summary](#summary)
-  - [Problem Statement](#problem-statement)
-  - [Proposal](#proposal)
-    - [Summary](#summary-1)
-    - [`ClusterPool.Spec.Inventory`](#clusterpoolspecinventory)
-    - [How To Use](#how-to-use)
-    - [Validation](#validation)
-    - [`Size` and `MaxSize`](#size-and-maxsize)
-    - [Pool Version](#pool-version)
-    - [Handling Inventory Updates](#handling-inventory-updates)
-      - [Adding An Inventory](#adding-an-inventory)
-      - [Adding An Entry to the Inventory](#adding-an-entry-to-the-inventory)
-      - [Removing An Entry from the Inventory](#removing-an-entry-from-the-inventory)
-      - [Deleting The Inventory](#deleting-the-inventory)
-    - [Maintaining the lease of the ClusterDeploymentCustomization](#maintaining-the-lease-of-the-clusterdeploymentcustomization)
-    - [Fairness](#fairness)
-  - [Future](#future)
-  - [Alternatives](#alternatives)
-    - [Bespoke Inventory Definition](#bespoke-inventory-definition)
-    - [Full Spec](#full-spec)
-    - [Hooks](#hooks)
+- [Summary](#summary)
+- [Problem Statement](#problem-statement)
+- [Proposal](#proposal)
+  - [Summary](#summary-1)
+  - [`ClusterPool.Spec.Inventory`](#clusterpoolspecinventory)
+  - [How To Use](#how-to-use)
+  - [Validation](#validation)
+  - [`Size` and `MaxSize`](#size-and-maxsize)
+  - [Pool Version](#pool-version)
+  - [Handling Inventory Updates](#handling-inventory-updates)
+    - [Adding An Inventory](#adding-an-inventory)
+    - [Adding An Entry to the Inventory](#adding-an-entry-to-the-inventory)
+    - [Removing An Entry from the Inventory](#removing-an-entry-from-the-inventory)
+    - [Deleting The Inventory](#deleting-the-inventory)
+  - [Maintaining the lease of the ClusterDeploymentCustomization](#maintaining-the-lease-of-the-clusterdeploymentcustomization)
+  - [Fairness](#fairness)
+- [Future](#future)
+- [Alternatives](#alternatives)
+  - [Bespoke Inventory Definition](#bespoke-inventory-definition)
+  - [Full Spec](#full-spec)
+  - [Hooks](#hooks)
 
 ## Summary
 
@@ -59,7 +58,7 @@ spec:
 
 and ClusterDeploymentCustomization CR will look like
 ```yaml
-apiVersion: v1
+apiVersion: hive.openshift.io/v1
 kind: ClusterDeploymentCustomization
 metadata:
   name: foo-cluster-deployment-customization
@@ -115,10 +114,10 @@ For the VSphere case, this allows the administrator to:
 - Create a ClusterDeploymentCustomization CR to patch `spec.metadata.name` field of the default install config generated by clusterpool controller. Please refer the section above of a sample CR. The content in `spec.installConfigPatches` field should be as follows
   ```yaml
   spec:
-  installConfigPatches:
-  - op: replace
-    path: metadata/name
-    value: foo
+    installConfigPatches:
+    - op: replace
+      path: metadata/name
+      value: foo
   ```
 - Add the name of ClusterDeploymentCustomization CR to `clusterPool.spec.inventory.ClusterDeploymentCustomizations` list. For ClusterDeploymentCustomization with a name `foo-cluster-deployment-customization` the clusterpool should be configured as follows
     ```yaml