For questions, doubts, guidances please use Discussions. Don't open a new Issue.

[carlosedp]

Since I don't have too many resources and time to address all questions regarding the deployments, the Issues section is a place to report problems or improvements to the stack.

This issue is a place where you can add a comment in case of a question where me or any community member can answer in a best effort manner.

If you deployed the monitoring stack and some targets are not available or showing no metrics in Grafana, make sure you don't have IPTables rules or use a firewall on your nodes before deploying Kubernetes.

If you don't want to receive further notifications, click "Unsubscribe" in the right bar, right above the participants list.

[YushchenkoAndrew]

I faced with an Issue, in which I couldn't open grafana and prometheus applications (link https://grafana.192.168.0.106.nip.io)

 $ curl http://prometheus.192.168.0.106.nip.io
 curl: (7) Failed to connect to prometheus.192.168.0.106.nip.io port 80: Connection refused
 $ curl https://prometheus.192.168.0.106.nip.io
 curl: (7) Failed to connect to prometheus.192.168.0.106.nip.io port 443: Connection refused

In the browser I got same Issue "Unable to connect".

I'm using k3s and I configured my master ip address
192.168.0.106 - it's a local ip address one of my workers node

I managed to successfully deploy all pods but I don't know how do I need to connect to the applications

 $ kubectl get ingress -n monitoring
 NAME                CLASS    HOSTS                               ADDRESS   PORTS     AGE
 alertmanager-main   <none>   alertmanager.192.168.0.106.nip.io             80, 443   54s
 grafana             <none>   grafana.192.168.0.106.nip.io                  80, 443   54s
 prometheus-k8s      <none>   prometheus.192.168.0.106.nip.io               80, 443   53s

 $ kubectl get pods -n monitoring
 NAME                                   READY   STATUS    RESTARTS   AGE
 prometheus-operator-6b8868d698-6xlvg   2/2     Running   0          14m
 arm-exporter-wmm6r                     2/2     Running   0          14m
 arm-exporter-67jpd                     2/2     Running   0          14m
 node-exporter-fbltt                    2/2     Running   0          14m
 alertmanager-main-0                    2/2     Running   0          14m
 arm-exporter-zhd5m                     2/2     Running   0          14m
 node-exporter-pzz6z                    2/2     Running   0          14m
 node-exporter-74fwt                    2/2     Running   0          14m
 grafana-7466bcc7c5-4hvpj               1/1     Running   0          14m
 kube-state-metrics-96bf99844-g9ssn     3/3     Running   0          14m
 prometheus-adapter-f78c4f4ff-kccbq     1/1     Running   0          14m
 prometheus-k8s-0                       3/3     Running   0          14m

Do you have any suggestions ?

[carlosedp]

You need to troubleshoot the access to your K3s cluster ingress that bridges the outside HTTP/HTTPS traffic to the pods.

Here is a reference: https://rancher.com/docs/k3s/latest/en/networking/

Have you deployed any application that uses HTTP(like NGINX, Apache) and been able to access it from your computer? It's similar to access Prometheus, Grafana and AlertManager.

[YushchenkoAndrew]

Yes, I created own blog site on JS but I didn't use ingress, I configured externalIP on Service, so... I will try to troubleshoot this issue.
Thanks for replay!

[YushchenkoAndrew]

I solve this issue. Thank for advice, at the end I just installed nginx, configured it and after that I was able to access to prometheus and grafana.
Thanks a lot!

[johnfried]

Love this project! I am unable to access prometheus.*.nip.io . I can access both Grafana and Alert manager. My ingress shows Prometheus, and is setup correctly. The one odd thing is when I look at all my pods in Monitoring ns; I do not have Prometheus-K8s (or something along those lines that I have seen in videos). The pods I have are Prometheus Adapter and Operator. I have Re-ran make vendor and deployed them. Same thing, again no errors anywhere. And also Prometheus-K8s has a service as I just looked. Does this make any sense? TIA

[exArax]

Is there a way to deploy grafana and prometheus pods to the master node only ? Because sometimes they are deployed to workers

[carlosedp]

@exArax You need to set your master nodes as schedulable. Even this way, Kubernetes can deploy the pods to other nodes. If you need to set to a specific set of nodes, you need pod affinity.

[carlosedp]

Love this project! I am unable to access prometheus.*.nip.io . I can access both Grafana and Alert manager. My ingress shows Prometheus, and is setup correctly. The one odd thing is when I look at all my pods in Monitoring ns; I do not have Prometheus-K8s (or something along those lines that I have seen in videos). The pods I have are Prometheus Adapter and Operator. I have Re-ran make vendor and deployed them. Same thing, again no errors anywhere. And also Prometheus-K8s has a service as I just looked. Does this make any sense? TIA

Doesn't make too much sense since the pods are created by the operator. Re-check your cluster and re-deploy the stack.

[johnfried]

I redeployed and all is well, thank you

[exArax]

@exArax You need to set your master nodes as schedulable. Even this way, Kubernetes can deploy the pods to other nodes. If you need to set to a specific set of nodes, you need pod affinity.

In case of grafana, I have to add the node affinity on the grafana-deployment.yaml that is inside the manifests folder, right?

[ClauNav]

Hello Carlos,
I've the same issue as YushchenkoAndrew.
I'm noob on Kubernetes (I built this cluster to learn about it)

The same issue on Alertmanager/Prometheus.

Could you please help me?

Thanks.

[carlosedp]

@exArax You need to set your master nodes as schedulable. Even this way, Kubernetes can deploy the pods to other nodes. If you need to set to a specific set of nodes, you need pod affinity.

In case of grafana, I have to add the node affinity on the grafana-deployment.yaml that is inside the manifests folder, right?

Yes, since the jsonnet code doesn't have the pod affinity for this.

[carlosedp]

Hello Carlos,
I've the same issue as YushchenkoAndrew.
I'm noob on Kubernetes (I built this cluster to learn about it)

The same issue on Alertmanager/Prometheus.

Could you please help me?

Thanks.

You need to make sure your Kubernetes cluster has an Ingress controller and can expose the applications. Check this first with something like an NGINX pod with a simple Hello World web page.

[Nenad13]

Hi Carlos,
Very cool project indeed. I am running Kubernetes on Ubuntu 20.04.1 (master) and a few of Raspberry Pi 4 (nodes) with raspbian on them. I installed Kubernetes with ansible playbook and it works fine.
I made all changes in vars.jsonnet as you suggested. The problem is after. make deploy I am getting this error:

root@asus:~/cluster-monitoring# make deploy
echo "Deploying stack setup manifests..."
Deploying stack setup manifests...
kubectl apply -f ./manifests/setup/
The connection to the server localhost:8080 was refused - did you specify the right host or port?
make: *** [Makefile:37: deploy] Error 1

Do you have any suggestions?

This is the configuration:
kubectl config view
apiVersion: v1
clusters:

• cluster:
  certificate-authority-data: DATA+OMITTED
  server: https://x.x.x.x:6443
  name: default
  contexts:
• context:
  cluster: default
  user: default
  name: default
  current-context: default
  kind: Config
  preferences: {}
  users:
• name: default
  user:
  password: xxxxxxx
  username: xxxxxxxx

Thank you in advance!

[ClauNav]

Hello Carlos,
I've the same issue as YushchenkoAndrew.
I'm noob on Kubernetes (I built this cluster to learn about it)

The same issue on Alertmanager/Prometheus.
Could you please help me?
Thanks.

You need to make sure your Kubernetes cluster has an Ingress controller and can expose the applications. Check this first with something like an NGINX pod with a simple Hello World web page.

Hello Carlos, You're right!
Thanks for taking your time replaying our newbies questions.

[carlosedp]

@ToMe25 @jevy the endpoints changed a bit in the past versions, might need adjusts in the k3s-overrides.jsonnet that creates the endpoints for them.

[braucktoon]

Hi,

Nice work!

Is it possible to add the speed-test-exporter (https://docs.miguelndecarvalho.pt/projects/speedtest-exporter) to this deployment? Seems like I need to add the exporter to the prometheus.yml but this file is buried in the prometheus-k8s-0 pod and not exposed. Any guidance if someone has already done it?

Thanks

[jamessewell]

Love the project - I'm wondering how I go about updating though?

I want to get AlertmanagerConfig - which isn't available in the vendored versions which are locked (I know you can use the secret).

Basically I just want to know if it's safe to bump versions and what I should be looking out for - I haven't had much luck so far.

[mrimp]

arm-exporter-rhtfh 0/2 ContainerCreating 0 27m arm-exporter-h4kr7 0/2 ContainerCreating 0 27m arm-exporter-g5hsd 0/2 ContainerCreating 0 27m node-exporter-fdxwm 2/2 Running 0 27m arm-exporter-b69l4 2/2 Running 0 27m arm-exporter-hfbjk 2/2 Running 0 27m node-exporter-8rm9g 2/2 Running 0 27m prometheus-adapter-585b57857b-b54k9 1/1 Running 0 27m node-exporter-ck5j9 2/2 Running 0 27m arm-exporter-mxhrb 2/2 Running 0 27m node-exporter-ct28t 2/2 Running 0 27m node-exporter-nh2cr 2/2 Running 0 27m node-exporter-7n5jc 2/2 Running 0 27m grafana-7bc4784744-q6xzp 1/1 Running 0 27m prometheus-operator-67755f959-gp44r 1/2 CrashLoopBackOff 9 27m kube-state-metrics-6cb6df5d4-sv9zv 2/3 CrashLoopBackOff 9 27m
raspberry pi 4 its a reinstall was working prior thx
ARM64

[assapir]

Hey, I am not sure where my problem is.

I am unable to connect to any of the ingresses.
My DNS is pi.hole server, which yields no answer for grafana.192.168.1.103.nip.io, so I manually added it as a DNS record, and I think it now returns the right answer:

dig  +short prometheus.192.168.1.103.nip.io
grafana.192.168.1.103.nip.io.
192.168.1.103

but opening the browser on that address fails.
I don't think I have DNS rebinding protection enabled.

All pods seems to be running:

❯ kubectl -n monitoring get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
node-exporter-hx9hs                   2/2     Running   2          49m   192.168.1.103   k3s-master    <none>           <none>
arm-exporter-r8xtd                    2/2     Running   2          50m   10.42.0.9       k3s-master    <none>           <none>
prometheus-operator-67755f959-vrwrk   2/2     Running   4          50m   10.42.1.12      k3s-node-01   <none>           <none>
arm-exporter-vnw7f                    2/2     Running   4          50m   10.42.1.11      k3s-node-01   <none>           <none>
prometheus-k8s-0                      3/3     Running   4          28m   10.42.1.13      k3s-node-01   <none>           <none>
node-exporter-dnrrb                   2/2     Running   4          49m   192.168.1.106   k3s-node-01   <none>           <none>
alertmanager-main-0                   2/2     Running   2          50m   10.42.3.7       k3s-node-03   <none>           <none>
node-exporter-fwk5c                   2/2     Running   2          49m   192.168.1.115   k3s-node-03   <none>           <none>
arm-exporter-9gqxz                    2/2     Running   2          50m   10.42.3.5       k3s-node-03   <none>           <none>
prometheus-adapter-585b57857b-xcp2j   1/1     Running   1          49m   10.42.3.6       k3s-node-03   <none>           <none>
arm-exporter-98npp                    2/2     Running   4          50m   10.42.2.10      k3s-node-02   <none>           <none>
kube-state-metrics-6cb6df5d4-przcx    3/3     Running   6          49m   10.42.2.8       k3s-node-02   <none>           <none>
node-exporter-p2w6r                   2/2     Running   4          49m   192.168.1.111   k3s-node-02   <none>           <none>
grafana-7bc4784744-68nzs              1/1     Running   2          49m   10.42.2.9       k3s-node-02   <none>           <none>

[mrimp]

Hey, I am not sure where my problem is.

I am unable to connect to any of the ingresses. My DNS is pi.hole server, which yields no answer for grafana.192.168.1.103.nip.io, so I manually added it as a DNS record, and I think it now returns the right answer:

dig  +short prometheus.192.168.1.103.nip.io
grafana.192.168.1.103.nip.io.
192.168.1.103

but opening the browser on that address fails. I don't think I have DNS rebinding protection enabled.

All pods seems to be running:

❯ kubectl -n monitoring get pods -o wide
NAME                                  READY   STATUS    RESTARTS   AGE   IP              NODE          NOMINATED NODE   READINESS GATES
node-exporter-hx9hs                   2/2     Running   2          49m   192.168.1.103   k3s-master    <none>           <none>
arm-exporter-r8xtd                    2/2     Running   2          50m   10.42.0.9       k3s-master    <none>           <none>
prometheus-operator-67755f959-vrwrk   2/2     Running   4          50m   10.42.1.12      k3s-node-01   <none>           <none>
arm-exporter-vnw7f                    2/2     Running   4          50m   10.42.1.11      k3s-node-01   <none>           <none>
prometheus-k8s-0                      3/3     Running   4          28m   10.42.1.13      k3s-node-01   <none>           <none>
node-exporter-dnrrb                   2/2     Running   4          49m   192.168.1.106   k3s-node-01   <none>           <none>
alertmanager-main-0                   2/2     Running   2          50m   10.42.3.7       k3s-node-03   <none>           <none>
node-exporter-fwk5c                   2/2     Running   2          49m   192.168.1.115   k3s-node-03   <none>           <none>
arm-exporter-9gqxz                    2/2     Running   2          50m   10.42.3.5       k3s-node-03   <none>           <none>
prometheus-adapter-585b57857b-xcp2j   1/1     Running   1          49m   10.42.3.6       k3s-node-03   <none>           <none>
arm-exporter-98npp                    2/2     Running   4          50m   10.42.2.10      k3s-node-02   <none>           <none>
kube-state-metrics-6cb6df5d4-przcx    3/3     Running   6          49m   10.42.2.8       k3s-node-02   <none>           <none>
node-exporter-p2w6r                   2/2     Running   4          49m   192.168.1.111   k3s-node-02   <none>           <none>
grafana-7bc4784744-68nzs              1/1     Running   2          49m   10.42.2.9       k3s-node-02   <none>           <none>

Try kubectl get ingress -n monitoring
if your getting this error it will never load.

Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

still exist today

[assapir]

How does one can create a custom scarper? I have my own metrics endpoint and I want Prometheus to scrape it, how can I add that?

[Cian911]

Hi,

Nice work!

Is it possible to add the speed-test-exporter (https://docs.miguelndecarvalho.pt/projects/speedtest-exporter) to this deployment? Seems like I need to add the exporter to the prometheus.yml but this file is buried in the prometheus-k8s-0 pod and not exposed. Any guidance if someone has already done it?

Thanks

I've created a new PR for just this here: #116 cc: @carlosedp

[Cian911]

It seems that the ksonnet-lib project which this project heavily utilises is no longer active and the project has since been archived.

I came across a newer project which has been kept up to date https://github.com/jsonnet-libs/k8s-libsonnet and gave it a shot porting this project over to using it, but with not much success (I'm not very familiar with jsonnet). By the looks of things it might be a bit of work to do it. Wondering what your thoughts are @carlosedp and if you might have the time to update?

[ToMe25]

@carlosedp I have locally updated all dependencies, once to the latest kube-prometheus version, and once to version 0.8, which is the last one that supports Kubernetes 1.20.
I have tweaked it until it runs, and collects some data, however a lot of data seems missing and many of the dashboards aren't working because of it.
Fixing this is something I am unfortunately unable to do.

Should I push this as a Draft PR as a starting point for someone else to continue, or discard it since I couldn't get it to work?
Also if I do push it, which of the two versions should I push?

[radicalgeek]

@ToMe25 @jevy the endpoints changed a bit in the past versions, might need adjusts in the k3s-overrides.jsonnet that creates the endpoints for them.

@carlosedp Any idea what the end points need to look like? It seems they now only respond when accessed over localhost/127.0.0.1 and not over the cluster ip for the manager (even locally). I tried to change the IP to 127.0.0.1 in the endpoint manifest for kube control manager but it won't allow it. Any idea how we can fix this?

[exArax]

Hi @carlosedp ,

I want to add in this stack a pod thati s actually an exporter for Kubevirt. The procedure requires to add a deployment file and service file for sure, but how would I configure Prometheus to scrape from kubevirt-prometheus-metrics? Which file should I edit?

Update If found that I don't need an exporter. Based on thishttps://kubevirt.io/user-guide/operations/component_monitoring/ I can integrate Kubevirt with Prometheus Operator. The thing is that I am applying to Kubevirt the namespace monitoring and the service account of the Operator (prometheus-operator) and I can't see the metrics that start with kubevirt_vmi.

[ToMe25]

@ToMe25 @jevy the endpoints changed a bit in the past versions, might need adjusts in the k3s-overrides.jsonnet that creates the endpoints for them.

@carlosedp Any idea what the end points need to look like? It seems they now only respond when accessed over localhost/127.0.0.1 and not over the cluster ip for the manager (even locally). I tried to change the IP to 127.0.0.1 in the endpoint manifest for kube control manager but it won't allow it. Any idea how we can fix this?

@radicalgeek I actually locally tweaked modules/k3s-overrides.jsonnnet until the generated manifests looked perfect to me, and the result was this:

So unfortunately I don't think I know enough about this issue to fix this.

I compared the generated manifests with those for kube-dns, and couldn't find any difference that I could explain this with.
This is my modified modules/k3s-overrides.jsonnet file in case you want to play around with it:

modules/k3s-overrides.jsonnet

local utils = import '../utils.libsonnet';
local vars = import '../vars.jsonnet';
local k = import 'ksonnet/ksonnet.beta.4/k.libsonnet';
local service = k.core.v1.service;
local servicePort = k.core.v1.service.mixin.spec.portsType;

{
  prometheus+:: {
    kubeControllerManagerPrometheusDiscoveryService:
      service.new('kube-controller-manager-prometheus-discovery', { 'k8s-app': 'kube-controller-manager' }, servicePort.newNamed('http-metrics', 10252, 10252)) +
      service.mixin.metadata.withNamespace('kube-system') +
      service.mixin.metadata.withLabels({ 'k8s-app': 'kube-controller-manager' }) +
      service.mixin.spec.withClusterIp('None'),

    kubeSchedulerPrometheusDiscoveryService:
      service.new('kube-scheduler-prometheus-discovery', { 'k8s-app': 'kube-scheduler' }, servicePort.newNamed('http-metrics', 10251, 10251)) +
      service.mixin.metadata.withNamespace('kube-system') +
      service.mixin.metadata.withLabels({ 'k8s-app': 'kube-scheduler' }) +
      service.mixin.spec.withClusterIp('None'),
  },
}

I also tried modifying the endpoint manifests to reference the right k8s-app instead of removing them, but that didn't fix it either.

[exArax]

Hi again,

I am deploying a VM through Kubevirt, which has cirros inside with node exporter. How can I add an endpoint for this exporter to Prometheus?

[carlosedp]

Hi all, I've just enabled Discussions on the project so we can have a proper forum for talking about it.

Please head to https://github.com/carlosedp/cluster-monitoring/discussions.

I've migrated this issue to a discussion but I suggest new topics to be opened instead of following on this one.

Thanks everyone!