Karpenter webhooks deleted from Argo UI but not API server

[andrewjamesbrown]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When upgrading from Karpenter 0.36.4 to 1.0.1, and prune enabled, the ValidatingWebhookConfiguration and MutatingWebhookConfiguration are removed from the Argo UI, but are not deleted from the Kubernetes API server - this requires manual intervention through kubectl to delete them

To Reproduce

Install Karpenter 0.36.4 as an ArgoCD Application, with prune enabled on the application
Upgrade to Karpenter 1.0.1.

Expected behavior
ValidatingWebhookConfiguration and MutatingWebhookConfiguration resources are removed from Kubernetes API Server as well as Argo UI.

Screenshots

Version
2.12.3

Paste the output from `argocd version` here.

Logs

Paste any relevant application logs here.

[andrewjamesbrown]

% k get mutatingwebhookconfiguration/defaulting.webhook.karpenter.k8s.aws -o yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  creationTimestamp: "2024-08-29T21:06:47Z"
  generation: 3
  labels:
    app.kubernetes.io/instance: karpenter
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: karpenter
    app.kubernetes.io/version: 0.36.4
    argocd.argoproj.io/instance: karpenter
    helm.sh/chart: karpenter-0.36.4
  name: defaulting.webhook.karpenter.k8s.aws
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Namespace
    name: kube-system
    uid: 61cb25f1-3bb9-4097-8d2e-0a97edee0782
  resourceVersion: "647618615"
  uid: c8b45dd0-dc04-4e7c-961f-148c3a443d41
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNSekNDQWUyZ0F3SUJBZ0lSQUtQWk95dWMzbTA4KzVPTFpjYXlNR0F3Q2dZSUtvWkl6ajBFQXdJd09qRVUKTUJJR0ExVUVDaE1MYTI1aGRHbDJaUzVrWlhZeElqQWdCZ05WQkFNVEdXdGhjbkJsYm5SbGNpNXJkV0psTFhONQpjM1JsYlM1emRtTXdIaGNOTWpRd09UQTBNak16TkRJMFdoY05NalF3T1RFeE1qTXpOREkwV2pBNk1SUXdFZ1lEClZRUUtFd3RyYm1GMGFYWmxMbVJsZGpFaU1DQUdBMVVFQXhNWmEyRnljR1Z1ZEdWeUxtdDFZbVV0YzNsemRHVnQKTG5OMll6QlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJGdEtrUzBVeDVFWnQya1kzbGFmckVuNgpwN3luZFY0ZnE4RmN2MnQ4RU9jSTFNbXM1OGpFRnJWT0YrUmdKT1hBV2ZsWlVDSG92bi8vSTRDOW44MWxjbTJqCmdkTXdnZEF3RGdZRFZSMFBBUUgvQkFRREFnS0VNQjBHQTFVZEpRUVdNQlFHQ0NzR0FRVUZCd01CQmdnckJnRUYKQlFjREFqQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUXJ1OWRWNUdPZXVQZFpFeVF0bmJMYgppTXFiTVRCdkJnTlZIUkVFYURCbWdnbHJZWEp3Wlc1MFpYS0NGV3RoY25CbGJuUmxjaTVyZFdKbExYTjVjM1JsCmJZSVphMkZ5Y0dWdWRHVnlMbXQxWW1VdGMzbHpkR1Z0TG5OMlk0SW5hMkZ5Y0dWdWRHVnlMbXQxWW1VdGMzbHoKZEdWdExuTjJZeTVqYkhWemRHVnlMbXh2WTJGc01Bb0dDQ3FHU000OUJBTUNBMGdBTUVVQ0lRQ0J0Q2xzSDM4WApaLzNYdFdVdGpYa3NoWGRQRU4wL3hudVlDdC9kOHNOWFpRSWdRK2ZkQWV0Mm51amJMSVJGNDVBYTErOWkxSi94CjAyUDh4VHpQUWo1bTJoUT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    service:
      name: karpenter
      namespace: kube-system
      path: /default/karpenter.k8s.aws
      port: 8443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: defaulting.webhook.karpenter.k8s.aws
  namespaceSelector:
    matchExpressions:
    - key: webhooks.knative.dev/exclude
      operator: DoesNotExist
  objectSelector: {}
  reinvocationPolicy: IfNeeded
  rules:
  - apiGroups:
    - karpenter.k8s.aws
    apiVersions:
    - v1beta1
    operations:
    - CREATE
    - UPDATE
    resources:
    - ec2nodeclasses
    - ec2nodeclasses/status
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

[andrewjamesbrown]

% k get validatingwebhookconfiguration/validation.webhook.config.karpenter.sh -o yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  creationTimestamp: "2024-08-29T21:06:47Z"
  generation: 3
  labels:
    app.kubernetes.io/instance: karpenter
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: karpenter
    app.kubernetes.io/version: 0.36.4
    argocd.argoproj.io/instance: karpenter
    helm.sh/chart: karpenter-0.36.4
  name: validation.webhook.config.karpenter.sh
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: Namespace
    name: kube-system
    uid: 61cb25f1-3bb9-4097-8d2e-0a97edee0782
  resourceVersion: "647618617"
  uid: 19ff993c-57be-4713-a590-a4cbe656dcc2
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNSekNDQWUyZ0F3SUJBZ0lSQUtQWk95dWMzbTA4KzVPTFpjYXlNR0F3Q2dZSUtvWkl6ajBFQXdJd09qRVUKTUJJR0ExVUVDaE1MYTI1aGRHbDJaUzVrWlhZeElqQWdCZ05WQkFNVEdXdGhjbkJsYm5SbGNpNXJkV0psTFhONQpjM1JsYlM1emRtTXdIaGNOTWpRd09UQTBNak16TkRJMFdoY05NalF3T1RFeE1qTXpOREkwV2pBNk1SUXdFZ1lEClZRUUtFd3RyYm1GMGFYWmxMbVJsZGpFaU1DQUdBMVVFQXhNWmEyRnljR1Z1ZEdWeUxtdDFZbVV0YzNsemRHVnQKTG5OMll6QlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJGdEtrUzBVeDVFWnQya1kzbGFmckVuNgpwN3luZFY0ZnE4RmN2MnQ4RU9jSTFNbXM1OGpFRnJWT0YrUmdKT1hBV2ZsWlVDSG92bi8vSTRDOW44MWxjbTJqCmdkTXdnZEF3RGdZRFZSMFBBUUgvQkFRREFnS0VNQjBHQTFVZEpRUVdNQlFHQ0NzR0FRVUZCd01CQmdnckJnRUYKQlFjREFqQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUXJ1OWRWNUdPZXVQZFpFeVF0bmJMYgppTXFiTVRCdkJnTlZIUkVFYURCbWdnbHJZWEp3Wlc1MFpYS0NGV3RoY25CbGJuUmxjaTVyZFdKbExYTjVjM1JsCmJZSVphMkZ5Y0dWdWRHVnlMbXQxWW1VdGMzbHpkR1Z0TG5OMlk0SW5hMkZ5Y0dWdWRHVnlMbXQxWW1VdGMzbHoKZEdWdExuTjJZeTVqYkhWemRHVnlMbXh2WTJGc01Bb0dDQ3FHU000OUJBTUNBMGdBTUVVQ0lRQ0J0Q2xzSDM4WApaLzNYdFdVdGpYa3NoWGRQRU4wL3hudVlDdC9kOHNOWFpRSWdRK2ZkQWV0Mm51amJMSVJGNDVBYTErOWkxSi94CjAyUDh4VHpQUWo1bTJoUT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    service:
      name: karpenter
      namespace: kube-system
      path: /validate/config.karpenter.sh
      port: 8443
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: validation.webhook.config.karpenter.sh
  namespaceSelector: {}
  objectSelector:
    matchLabels:
      app.kubernetes.io/part-of: karpenter
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - configmaps/*
    scope: Namespaced
  sideEffects: None
  timeoutSeconds: 10

[andrewjamesbrown]

Possibly related to knative/serving#15483

[andrewjamesbrown]

aws/karpenter-provider-aws#6847 (comment)

[Adityashar]

When upgrading from Karpenter 0.36.4 to 1.0.1, and prune enabled, the ValidatingWebhookConfiguration and MutatingWebhookConfiguration

@andrewjamesbrown Does Karpenter still deploy webhooks in 0.36? I remember reading somewhere that the webhooks were removed from Karpenter in 0.27.x