[o11y] Add default Prometheus Rule to alert off Firefly core metrics

[calbritt]

Alerts if the firefly core target is not up.

Alerts if any actions below get rejected:

• mints
• transfers
• burns
• private messages
• broadcasts

Resolving this issue - #47

[onelapahead]

This is great thanks for tackling this one @calbritt !

I took at a glance at ingress-nginx for some community guidance and inspiration: https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L708-L745

I'll only ask we refactor to have similar pattern of rules but having that be a templatable multi-line string (similar to what we've done for extraContainers for example) rather than a YAML list.

And then the default could be something like:

rules: |
  {{- include "firefly.defaultAlertingRules" . | nindent ... }}

And we move these recommended rules you've made into a helper function. That way folks can reuse our rules and their own if the want, or just completely replace them.

The helper function can refer to values like warningSeverity and other helpers too just like a template which is nice.

[onelapahead]

Let's also either add a new ci/ test or update an existing one to include this templating path where the rules are enabled with the default alerts.

[onelapahead]

Suggested change

	rules: |
	{{- include "firefly.defaultPrometheusRules" . | nindent 4 }}
	{{- if .Values.core.metrics.prometheusRule.rules }}
	{{- tpl .Values.core.metrics.prometheusRule.rules . | nindent 4 }}
	{{- end }}
	rules:
	{{- tpl .Values.core.metrics.prometheusRule.rules . | nindent 4 }}

[onelapahead]

Note the rules needs to be YAML not a multi-line string, think we just got it criss crossed between here and the values.

[onelapahead]

Suggested change

	rules: []
	rules: |
	{{- include "firefly.defaultPrometheusRules" . }}

[onelapahead]

Note this must be a string if passed to tpl, and we don't need to indent because its handled for us when we template the string within the PrometheusRule YAML.

This will allow users to entirely override the rules if they want which is preferred.

[nguyer]

Closing this PR due to inactivity. Please feel free to re-open if this work is still ongoing.