Alertmanager/ Prometheus Rule #2572

danielstankw · 2024-12-18T15:02:26Z

Hi all,
I have been breaking my head for past few days on the following:

I have a prometheusRule and AlermanagerConfig CRDs. And promtheus/ thanos installed using kube-prometheus
When I deploy them in K8s, I can see them in the UI

The issue is that only one of them send me email notification, and the other doesn't. Also, even if it does, it doesnt send email consistently.
How can I fix that?

My prometheusRule

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  labels:
    release: monitoring
  name: daniel-custom-rules-prod
  namespace: monitoring
spec:
  groups:
    - name: "pod-error-rule"
      rules:
        - alert: PodErrorAlert
          for: 3m
          expr: count(kube_pod_container_status_terminated_reason{reason='Error'}) by (namespace, pod)
          labels:
            severity: critical
          annotations:
            summary: Pod has been in error state for more than 10 minutes.
            message: "pod {{ $labels.pod }} in the {{ $labels.namespace }} namespace has been in error state for more than 10m"
    - name: "pod-pending-rule"
      rules:
        - alert: PodPendingFor30mAlert
          for: 2m
          expr: count(kube_pod_status_phase{phase="Pending"} > 0) by (namespace, pod)
          labels:
            severity: critical
          annotations:
            summary: Pod has been pending for more than 30 minutes.
            message: "pod {{ $labels.pod }} in the {{ $labels.namespace }} namespace has been in pending state for more than 30m"

My alertmanagerConfig

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alert-config
  namespace: monitoring
  labels:
    release: monitoring
spec:
  route:
    groupBy: ['cluster', 'namespace']
    groupWait: 30s
    groupInterval: 10m
    repeatInterval: 10m
    receiver: email
  receivers:
    - name: email
      emailConfigs:
        - to: 'XXXX@XXX.com'
          from: 'YYYYYY@XXX.com'
          smarthost: 'ZZZZZ.XXXX.com:25'
          headers:
            - key: subject
              value: "[{{ .Status | toUpper }}] Alert Notification"
          requireTLS: false
          sendResolved: false
          html: |
            <html>
            <body>
              <h3>Alerts Summary</h3>       
              <ul>
              {{ range .Alerts }}
                <li>
                  <b>Alert:</b> {{ .Annotations.summary }}<br/>
                  <b>Description:</b> {{ .Annotations.message }}<br/>
                  <b>Environment:</b> {{ .Labels.env }}<br/>
                  <b>Cluster:</b> {{ .Labels.cluster }}<br/>
                  <b>Namespace:</b> {{ .Labels.namespace }}<br/>
                  <b>Pod:</b> {{ .Labels.pod }}<br/>
                  <b>Severity:</b> {{ .Labels.severity }}<br/>
                  <b>Status:</b> {{ .Status | toUpper }}<br/>
                  <b>Starts At:</b> {{ .StartsAt }}<br/>
                  {{ if eq .Status "resolved" }}
                  <b>Resolved At:</b> {{ .EndsAt }}<br/>
                  {{ end }}
                  <!-- Grafana Button -->
                  <a href="https://my-grafana.com/d/ddqn51m3mrn5sa/cluster-view?orgId=1&var-cluster={{ .Labels.cluster }}&var-resolution=30s" 
                    style="display:inline-block;padding:10px 20px;color:black;background-color:#d3d3d3;border-radius:5px;text-decoration:none;font-weight:bold;">
                    Click to view in Grafana
                  </a>
                </li>
              {{ end }}
              </ul>
            </body>
            </html>

danielstankw added the kind/support label Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alertmanager/ Prometheus Rule #2572

Alertmanager/ Prometheus Rule #2572

danielstankw commented Dec 18, 2024 •

edited

Loading

Alertmanager/ Prometheus Rule #2572

Alertmanager/ Prometheus Rule #2572

Comments

danielstankw commented Dec 18, 2024 • edited Loading

danielstankw commented Dec 18, 2024 •

edited

Loading