KubeClientCertificateExpiration warning alert from 7 days to 2 days for AKS running on Azure stack HCI

[dadimah]

Hi,

I would like to change the below alert from 7 days to 2 days since AKS running on Azure stack HCI nodes will rotate the Client certificate every 4 days. In that case below condition is invalid for AKS-HCI Kubernetes.

We're using the helm chart to deploy the Kube-Prometheus stack and all Prometheus rules are attached to POD via configmap of prometheus rules. is there any way we can override this alert condition locally? Please suggest.

Helm Chart Repo: https://prometheus-community.github.io/helm-charts'

Alert:

• name: kubernetes-system-apiserver
  rules:
  • alert: KubeClientCertificateExpiration
    annotations:
    description: A client certificate used to authenticate to the apiserver is expiring in less than 7.0 days.
    runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration
    summary: Client certificate is about to expire.
    expr: apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 604800
    labels:
    severity: warning

Alert firing in prometheus

[metalmatze]

You can change those in the provided config when using the mixin via jsonnet:

kubernetes-mixin/alerts/kube_apiserver.libsonnet

Lines 9 to 10 in 3f040a8

	certExpirationWarningSeconds: 7 * 24 * 3600,
	certExpirationCriticalSeconds: 1 * 24 * 3600,

If this isn't possible via some Helm chart then you'd need to raise an issue with those maintainers 😉

[github-actions]

This issue has not had any activity in the past 30 days, so the
stale label has been added to it.

• The stale label will be removed if there is new activity
• The issue will be closed in 7 days if there is no new activity
• Add the keepalive label to exempt this issue from the stale check action

Thank you for your contributions!