Error executing query: multiple matches for labels: grouping labels must ensure unique matches

[usternik]

We are seeing the following errors from Alertmanager.
Prometheus monitoring/prometheus-prometheus-operator-kube-p-prometheus-1 has failed to evaluate 4 rules in the last 5m.

It appears to be happening every few hours, for a few minutes. Here is an image for reference:

Looking at prometheus logs we see:
ts=2022-06-26T13:48:58.529Z caller=manager.go:609 level=warn component="rule manager" group=k8s.rules msg="Evaluating rule failed" rule="record: node_namespace_pod_container:container_memory_rss\nexpr: container_memory_rss{image!=\"\",job=\"kubelet\",metrics_path=\"/metrics/cadvisor\"}\n * on(namespace, pod) group_left(node) topk by(namespace, pod) (1, max by(namespace,\n pod, node) (kube_pod_info{node!=\"\"}))\n" err="multiple matches for labels: grouping labels must ensure unique matches"

I am not sure if it's a specific thing in our cluster that triggers those errors but it looks like node_namespace_pod_container:container_memory_rss causes these errors. This is the query: container_memory_rss{image!="",job="kubelet",metrics_path="/metrics/cadvisor"} * on(namespace, pod) group_left(node) topk by(namespace, pod) (1, max by(namespace, pod, node) (kube_pod_info{node!=""}))

[usternik]

We are running on AKS with a substantial amount of spot machines. I suspect it originates in unready Nodes but not sure

[github-actions]

This issue has not had any activity in the past 30 days, so the
stale label has been added to it.

• The stale label will be removed if there is new activity
• The issue will be closed in 7 days if there is no new activity
• Add the keepalive label to exempt this issue from the stale check action

Thank you for your contributions!