Support patching installer manifests

[2uasimojo]

With this change, you can use new API field ClusterDeployment.Spec.Provisioning.CustomizationRef to point to a ClusterDeploymentCustomization (hereinafter "CDC") object in the same namespace as the ClusterDeployment (CD).

ClusterDeploymentCustomizations:
CDC accepts a new subfield, Spec.InstallerManifestPatches, which consists of:

• Glob: a string representing a file glob, relative to the installer working directory, matching one or more manifest files.
• Patches: a list of PatchEntity representing RFC6902 JSON patches to apply to the matched manifest(s).

Also, I got really annoyed having to type out clusterdeploymentcustomizations on the CLI, so I added abbreviation cdc to the schema.

ClusterPools:
CDC was already being used by ClusterPool-owned CDs to allow patching the install-config generated from the template referred to by ClusterPool.Spec.InstallConfigSecretTemplateRef. With this change, ClusterPool-owned CDs can start using manifest patches in two ways (not mutually exclusive):

• Patches specific to the CD can be included in the InstallerManifestPatches field of the existing Inventory CDCs.
• Patches applicable to all CDs in the pool can be provided by a CDC referenced via a new ClusterPool.Spec.CustomizationRef field.

HIVE-1793

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 2uasimojo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [2uasimojo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[2uasimojo]

/test e2e

[2uasimojo]

This is going well:

efried@efried-thinkpadp16vgen1:~/go/src/github.com/openshift/hive$ oc get cd efried416 -o yaml | yq r - spec.provisioning.customizationRef
name: mycdc

mycdc looks like:

apiVersion: hive.openshift.io/v1
kind: ClusterDeploymentCustomization
metadata:
  name: mycdc
  namespace: efried
spec:
  installerManifestPatches:
  - manifestSelector:
      glob: cluster-api/*/*machine*.yaml
    patches:
    - op: add
      path: /metadata/labels
      valueJSON: |
        {"efried.openshift.io/foo": "bar"}

The installmanager logs include:

time="2024-11-09T23:44:50Z" level=info msg="Found 1 InstallerManifestPatch entries from ClusterDeployment.Spec.CustomizationRef mycdc" installID=qgwtrjmj
time="2024-11-09T23:44:50Z" level=info msg="manifest patch glob 0 (cluster-api/*/*machine*.yaml) matched 8 files" installID=qgwtrjmj
time="2024-11-09T23:44:50Z" level=info msg="patching manifest /output/cluster-api/machines/10_inframachine_efried416-ttb8s-bootstrap.yaml" installID=qgwtrjmj
time="2024-11-09T23:44:50Z" level=info msg="patching manifest /output/cluster-api/machines/10_inframachine_efried416-ttb8s-master-0.yaml" installID=qgwtrjmj
time="2024-11-09T23:44:50Z" level=info msg="patching manifest /output/cluster-api/machines/10_inframachine_efried416-ttb8s-master-1.yaml" installID=qgwtrjmj
time="2024-11-09T23:44:50Z" level=info msg="patching manifest /output/cluster-api/machines/10_inframachine_efried416-ttb8s-master-2.yaml" installID=qgwtrjmj
time="2024-11-09T23:44:50Z" level=info msg="patching manifest /output/cluster-api/machines/10_machine_efried416-ttb8s-bootstrap.yaml" installID=qgwtrjmj
time="2024-11-09T23:44:50Z" level=info msg="patching manifest /output/cluster-api/machines/10_machine_efried416-ttb8s-master-0.yaml" installID=qgwtrjmj
time="2024-11-09T23:44:50Z" level=info msg="patching manifest /output/cluster-api/machines/10_machine_efried416-ttb8s-master-1.yaml" installID=qgwtrjmj
time="2024-11-09T23:44:50Z" level=info msg="patching manifest /output/cluster-api/machines/10_machine_efried416-ttb8s-master-2.yaml" installID=qgwtrjmj

and a sample manifest in the in-progress provision pod was correctly patched:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Machine
metadata:
  creationTimestamp: null
  labels:
    efried.openshift.io/foo: bar    # <=== L@@K
  name: efried416-ttb8s-master-0
spec:
  bootstrap:
    dataSecretName: efried416-ttb8s-master
  clusterName: efried416-ttb8s
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSMachine
    name: efried416-ttb8s-master-0
status:
  bootstrapReady: false
  infrastructureReady: false

[2uasimojo]

I just need to test the clusterpool path (and maybe noodle how the error conditions manifest) and we'll be good to land this.

[2uasimojo]

/cc @abraverm

[abraverm]

In Clusterpool we do reservation of the CDC, should this mechanism somewhat migrate/copied to CD controller?

[2uasimojo]

In Clusterpool we do reservation of the CDC, should this mechanism somewhat migrate/copied to CD controller?

For pool inventory, reserving CDCs makes sense because they're intended to enable exclusive/unique settings such as reserved IP addresses. And manifest-patching CDCs used for clusterpool inventory will still be subject to reservation as usual. But for this use case -- CDs' CustomizationRef-named CDCs -- I think we explicitly want manifest patching to be usable by multiple CDs without restriction. Does that make sense?

[codecov]

Codecov Report

Attention: Patch coverage is 55.80110% with 80 lines in your changes missing coverage. Please review.

Project coverage is 42.90%. Comparing base (3ccd44d) to head (4239df0).

Files with missing lines	Patch %	Lines
pkg/installmanager/installmanager.go	14.70%	28 Missing and 1 partial ⚠️
.../clusterdeployment/clusterdeployment_controller.go	32.43%	25 Missing ⚠️
pkg/controller/utils/clusterdeployment.go	56.25%	13 Missing and 1 partial ⚠️
...g/controller/clusterpool/clusterpool_controller.go	73.33%	11 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2499      +/-   ##
==========================================
- Coverage   46.26%   42.90%   -3.36%     
==========================================
  Files         279      279              
  Lines       32913    33075     +162     
==========================================
- Hits        15227    14191    -1036     
- Misses      16406    17691    +1285     
+ Partials     1280     1193      -87     

Files with missing lines	Coverage Δ
pkg/controller/clusterpool/collections.go	75.74% <100.00%> (+0.16%)	⬆️
pkg/controller/utils/sa.go	67.90% <ø> (ø)
...entcustomization/clusterdeploymentcustomization.go	93.33% <100.00%> (+1.12%)	⬆️
pkg/test/clusterpool/clusterpool.go	95.23% <100.00%> (+0.18%)	⬆️
...shift/hive/apis/hive/v1/clusterdeployment_types.go	0.00% <ø> (ø)
...is/hive/v1/clusterdeploymentcustomization_types.go	0.00% <ø> (ø)
...m/openshift/hive/apis/hive/v1/clusterpool_types.go	0.00% <ø> (ø)
...g/controller/clusterpool/clusterpool_controller.go	58.01% <73.33%> (+1.35%)	⬆️
pkg/controller/utils/clusterdeployment.go	68.21% <56.25%> (-3.22%)	⬇️
.../clusterdeployment/clusterdeployment_controller.go	30.52% <32.43%> (-36.51%)	⬇️
... and 1 more

... and 10 files with indirect coverage changes

[openshift-ci]

@2uasimojo: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.