Unable to access the Prometheus metrics of kube-scheduler and kube-control-manager in microk8s cluster

[developer1622]

Summary

I have a microk8s 2-node cluster(one is the control plane and another is worker node)

I can access Prometheus metrics following components

API Server
kube-proxy
dns

Not able to access Prometheus metrics of the following components

Kube-scheduler
Kube-contol-manager

What Should Happen Instead?

As per this official doc: https://microk8s.io/docs/services-and-ports they run on 10259 and 10257

expectation is that able to access the metrics of scheduler and controller using standard Prometheus scrape configuration , but I am not able to.

      - job_name: kube-scheduler
        honor_labels: true
        scrape_interval: 2m
        scrape_timeout: 55s
        metrics_path: /metrics
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        scheme: https
        tls_config:
          insecure_skip_verify: true
        static_configs:
          - targets: ["127.0.0.1:10259"]

      - job_name: kube-controller-manager
        scrape_interval: 2m
        scrape_timeout: 55s
        metrics_path: /metrics
        scheme: https
        static_configs:
          - targets: ["127.0.0.1:10257"]
        honor_labels: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        tls_config:
          insecure_skip_verify: true

But is not working even though my service account has cluster-admin role , when I access above ports, I am getting 403, tried with certificates also, please find below

curl -k -H "Authorization: Bearer $token" curl https://127.0.01:10257/metrics
curl: (6) Could not resolve host: curl
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403

Same goes with the Kube-scheduler also

Reproduction Steps

I have attached in the comment section to reproduce.

Introspection Report

I am not allowed to copy cluster data

Can you suggest a fix?

I am not sure but it is good to have seperate metrics for each K8s component on their own dedicated port.

Are you interested in contributing with a fix?

No, I do not have skillset

Thank you.

[developer1622]

In Prometheus

To fetch metrics from any component, we need host address and port number
and then access permission

I have

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: otelcontribcol
  name: otelcontribcol
  namespace: default
---
# apiVersion: rbac.authorization.k8s.io/v1
# kind: ClusterRole
# metadata:
#   labels:
#     app: otelcontribcol
#   name: otelcontribcol
# rules:
#   - nonResourceURLs:
#       - /version
#       - /healthz
#       - /metrics
#     verbs:
#       - get
#   - apiGroups:
#       - ""
#     resources:
#       - "*"
#     verbs:
#       - get
#       - list
#       - watch
#   - apiGroups:
#       - apps
#     resources:
#       - daemonsets
#       - deployments
#       - replicasets
#       - statefulsets
#     verbs:
#       - get
#       - list
#       - watch
#   - apiGroups:
#       - extensions
#     resources:
#       - daemonsets
#       - deployments
#       - replicasets
#     verbs:
#       - get
#       - list
#       - watch
#   - apiGroups:
#       - batch
#     resources:
#       - jobs
#       - cronjobs
#     verbs:
#       - get
#       - list
#       - watch
#   - apiGroups:
#       - autoscaling
#     resources:
#       - horizontalpodautoscalers
#     verbs:
#       - get
#       - list
#       - watch
#   - apiGroups:
#       - metrics.k8s.io
#     resources:
#       - pods
#       - persistentvolumeclaims
#       - namespaces
#       - networks
#     verbs:
#       - get
#       - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: otelcontribcol
  name: otelcontribcol
rules:
  - apiGroups:
      - "*"
    resources:
      - "*"
    verbs:
      - "*"
  - nonResourceURLs:
      - "*"
    verbs:
      - "*"

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: otelcontribcol
  name: otelcontribcol
roleRef:
  # apiGroup: rbac.authorization.k8s.io
  # kind: ClusterRole
  # name: otelcontribcol

  # for debugging, we can try with cluster-admin also
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: otelcontribcol
    namespace: default

And the Prometheu scrape config mentioned below.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  labels:
    app: prometheus
data:
  prometheus.yml: |
    global:
      scrape_interval: 2m
      evaluation_interval: 2m
    scrape_configs:
      - job_name: kube-scheduler
        honor_labels: true
        scrape_interval: 2m
        scrape_timeout: 55s
        metrics_path: /metrics
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        scheme: https
        tls_config:
          insecure_skip_verify: true
        static_configs:
          - targets: ["127.0.0.1:10259"]

      - job_name: kube-controller-manager
        scrape_interval: 2m
        scrape_timeout: 55s
        metrics_path: /metrics
        scheme: https
        static_configs:
          - targets: ["127.0.0.1:10257"]
        honor_labels: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        tls_config:
          insecure_skip_verify: true

[neoaggelos]

Hi @developer1622

127.0.0.1:10257 is the right address from the networking pov of the host, but not from the prometheus pod.

You should use the IP address of the node instead (which you can find with "microk8s kubectl get node -o wide". The port numbers look fine

[developer1622]

Hi @neoaggelos , thank you for responding

Yes, I have hostNetwork: true in pod settings, so it should be able to access hostNetwork

moreover, I have also tried on the node where my control plane running

in both the cases, I am not able to access it

Could you please provide any doc ref that shows we can access kube-scheduler and kube-control -manger metrics through Prometheus?

Thank you

[developer1622]

Hi @neoaggelos as suggested,
I execed into the Pod and used the curl and my service account has cluster admin access, but still I am getting 403 error.

$ curl -k -H "Authorization: Bearer $token" curl https://<my-node-ipv4-address>:10257/metrics
curl: (6) Could not resolve host: curl
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/metrics\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}~ $ 

Thank you

[neoaggelos]

Hi @developer1622, ah yes, you are right, my bad.

Have a look at what we do for the observability addon, which deploys kube-prom-stack on MicroK8s.

https://github.com/canonical/microk8s-core-addons/blob/ad40d154f9d418201e49e18b2cd02ebdab94ac24/addons/observability/enable#L106-L114

You need to specify these arguments on the services (arguments files can be found in /var/snap/microk8s/current/args/...). Then, the kube-controller-manager and kube-scheduler will be able to recognise the service account tokens and the metrics endpoints should work

[developer1622]

Thank you for responding in Slack and here @neoaggelos.