[bug] Some dashboards are broken on high pods count

[maxpain]

Describe the bug

Hello. We have a lot of short-lived pods in our clusters. It's also a problem for frequent CronJobs.

Screen.Recording.2024-11-09.at.10.35.49.mov

How to reproduce?

No response

Expected behavior

No response

Additional context

No response

[EladAviczer]

Have you tried increasing the CPU resources allocated to Prometheus?

[maxpain]

@EladAviczer did you watch the video?

[EladAviczer]

*victoriaMetrics

[maxpain]

*victoriaMetrics

The problem is not in Prometheus/VictoriaMetrics, but in the grafana dashboard itself.

[EladAviczer]

The dashboard uses VictoriaMetrics to query the data, you get 422 Unprocessable Content error when calling the promql/metricsQL query.

I don't say that i'm 100% sure that it is a VictoriaMetrics problem but it could be and you should check it too. : )

[maxpain]

The dashboard uses VictoriaMetrics to query the data, you get 422 Unprocessable Content error when calling the promql/metricsQL query.

The problem is that there are a lot of pods (because CronJob running every minute), and this dashboard tries to pass the array of pod names (1440 pods for last 24 hours), which will fail on any installation (Prometheus or VictoriaMetrics)

[dotdc]

Hi @maxpain,

The created_by variable was introduced to enable filtering on deployments, but if there are too many pods, you'll up end with a 422 Unprocessable Entity error as you just experienced.

I’ll check if there’s a better solution, but removing the created_by variable might work better in your case.