using invalid client_id will keep on populating more and more metrics

[cboitel]

Description

Since metrics do use client_id as label, use of an invalid value can be used to create more and more metrics causing prometheus collectors to fail.

This has multiple possible side effects:

1. it can be used as a mean to disrupt telemetry collection prior to some attack.
2. you can slow create more metrics and enforce the system to perform more GC than usual and even lead to outofmemory in extreme cases

Expected Behavior

To be defined but one could:

1. allow to disable client_id labeling and make it the default behaviour: documentation would warn about the issue
2. allow to provide a whitelist of valid clients (per provider if possible)
3. allow to define some delay after which a metric with a no longer used client_id label would be removed from exported data

Actual Behavior

4 new lines in prometheus exported data which will be forever (for ever)

Ex:

keycloak_login_attempts_created{client_id="wrong-XXXX"...}
keycloak_login_attempts_total{client_id="wrong-XXXX"...}
keycloak_failed_login_attempts_total{client_id="wrong-XXXX",error="client_not_found",provider="keycloak",realm="users"}
keycloak_failed_login_attempts_created{client_id="wrong-XXXX",error="client_not_found",provider="keycloak",realm="users"}

More and more lines returned means:

1. more and more data to store in prometheus
2. collector will end-up crashing since it requires more and more memory to collect.

Environment

Based on keycloak docker image 24.0.1-0 with metrics spi 5.0.0

Steps to reproduce

Simply initiate an OIDC/OAUTH2 flow against Keycloak using an unknown client_id which can be different each time.