How to collect labels and annotations of Kubernetes objects

1. Why kube-status-metrics is needed

Kubernetes monitoring mainly focuses on two kinds of metrics:

Basic performance metrics

CPU, memory, disk, network, and so on. These can be collected by deploying node-exporter as a DaemonSet and having Prometheus scrape the relevant metrics.

Resource object metrics

The replica count of a Deployment, the running state of a Pod, and so on. These metrics require kube-status-metrics to poll the Kubernetes API, query them, and expose them before Prometheus can see them.

2. What metrics kube-status-metrics provides by default

The metric categories include:

CertificateSigningRequest Metrics
ConfigMap Metrics
CronJob Metrics
DaemonSet Metrics
Deployment Metrics
Endpoint Metrics
Horizontal Pod Autoscaler Metrics
Ingress Metrics
Job Metrics
Lease Metrics
LimitRange Metrics
MutatingWebhookConfiguration Metrics
Namespace Metrics
NetworkPolicy Metrics
Node Metrics
PersistentVolume Metrics
PersistentVolumeClaim Metrics
Pod Disruption Budget Metrics
Pod Metrics
ReplicaSet Metrics
ReplicationController Metrics
ResourceQuota Metrics
Secret Metrics
Service Metrics
StatefulSet Metrics
StorageClass Metrics
ValidatingWebhookConfiguration Metrics
VerticalPodAutoscaler Metrics
VolumeAttachment Metrics

Taking Pod as an example:

kube_pod_annotations
kube_pod_info
kube_pod_ips
kube_pod_start_time
kube_pod_completion_time
kube_pod_owner
kube_pod_labels
kube_pod_nodeselectors
kube_pod_status_phase
kube_pod_status_ready
kube_pod_status_scheduled
kube_pod_containeHnfo
kube_pod_container_status_waiting
kube_pod_container_status_waiting_reason
kube_pod_container_status_running
kube_pod_container_state_started
kube_pod_container_status_terminated
kube_pod_container_status_terminated_reason
kube_pod_container_status_last_terminated_reason
kube_pod_container_status_ready
kube_pod_container_status_restarts_total
kube_pod_container_resource_requests
kube_pod_container_resource_limits
kube_pod_overhead_cpu_cores
kube_pod_overhead_memory_bytes
kube_pod_runtimeclass_name_info
kube_pod_created
kube_pod_deletion_timestamp
kube_pod_restart_policy
kube_pod_init_container_info
kube_pod_init_container_status_waiting
kube_pod_init_container_status_waiting_reason
kube_pod_init_container_status_running
kube_pod_init_container_status_terminated
kube_pod_init_container_status_terminated_reason
kube_pod_init_container_status_last_terminated_reason
kube_pod_init_container_status_ready
kube_pod_init_container_status_restarts_total
kube_pod_init_containerLresource_limits
kube_pod_init_container^resource_requests
kube_pod_spec_volumes_persistentvolumeclaims_info
kube_pod_spec_volumes_persistentvolumeclaims_readonly
kube_pod_status_reason
kube_pod_status_scheduled_time
kube_pod_status_unschedulable

The related metrics are very rich, and they are basically enough to observe the running state of Kubernetes.

3. How to scrape labels and annotations

By default, the kube_pod_labels and kube_pod_annotations metrics contain only the name and namespace labels.

If you need to monitor more labels and annotations, you need to use two startup parameters of kube-status-metrics: --metric-labels-allowlist and --metric-annotations-allowlist.

Note that older versions of kube-status-metrics do not fully support these two parameters. The configuration below uses version 2.4.2.

1
2
3
4
5
6
7
8
      containers:
      - args:
        - --port=8080
        - --metric-labels-allowlist=pods=[*]
        - --metric-annotations-allowlist=pods=[*]
        - --resources=certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments
        - --telemetry-port=8081
        image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.4.2

Prepare a Pod as the observation target

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
kubectl -n tekton-pipelines get pod tekton-pipelines-controller-6f449d874b-mc7nl -o yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
    cni.projectcalico.org/containerID: 8a505a530b501ad80ce471e86b553257e4ec3541313bc4245233f60a04dd3619
    cni.projectcalico.org/podIP: 10.233.105.3/32
    cni.projectcalico.org/podIPs: 10.233.105.3/32
  creationTimestamp: "2022-04-06T01:44:20Z"
  generateName: tekton-pipelines-controller-6f449d874b-
  labels:
    app: tekton-pipelines-controller
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: default
    app.kubernetes.io/name: controller
    app.kubernetes.io/part-of: tekton-pipelines
    app.kubernetes.io/version: v0.24.1
    pipeline.tekton.dev/release: v0.24.1
    pod-template-hash: 6f449d874b
    version: v0.24.1
  name: tekton-pipelines-controller-6f449d874b-mc7nl

Observe kube_pod_labels

Before enabling the kube-status-metrics switch

After enabling the kube-status-metrics switch

kube_pod_labels{app_kubernetes_io_component=“metrics”, app_kubernetes_io_instance=“prometheus”, app_kubernetes_io_managed_by=“Helm”, app_kubernetes_io_name=“kube-state-metrics”, app_kubernetes_io_part_of=“kube-state-metrics”, app_kubernetes_io_version=“2.3.0”, exported_namespace=“tekton-pipelines”, helm_sh_chart=“kube-state-metrics-4.4.3”, instance=“10.233.105.11:8080”, job=“kubernetes-service-endpoints”, label_app=“tekton-pipelines-controller”, label_app_kubernetes_io_component=“controller”, label_app_kubernetes_io_instance=“default”, label_app_kubernetes_io_name=“controller”, label_app_kubernetes_io_part_of=“tekton-pipelines”, label_app_kubernetes_io_version=“v0.24.1”, label_pipeline_tekton_dev_release=“v0.24.1”, label_pod_template_hash=“6f449d874b”, label_version=“v0.24.1”, namespace=“monitor”, node=“node4”, pod=“tekton-pipelines-controller-6f449d874b-mc7nl”, service=“prometheus-kube-state-metrics”, uid=“412f8383-1c5c-4f61-8198-453bdb204911”}

Many labels starting with label_ will be added.

kube_pod_annotations

Before enabling the kube-status-metrics switch

kube_pod_annotations{app_kubernetes_io_component=“metrics”, app_kubernetes_io_instance=“prometheus”, app_kubernetes_io_managed_by=“Helm”, app_kubernetes_io_name=“kube-state-metrics”, app_kubernetes_io_part_of=“kube-state-metrics”, app_kubernetes_io_version=“2.3.0”, exported_namespace=“tekton-pipelines”, helm_sh_chart=“kube-state-metrics-4.4.3”, instance=“10.233.96.11:8080”, job=“kubernetes-service-endpoints”, namespace=“monitor”, node=“node2”, pod=“tekton-pipelines-controller-6f449d874b-mc7nl”, service=“prometheus-kube-state-metrics”, uid=“412f8383-1c5c-4f61-8198-453bdb204911”}

After enabling the kube-status-metrics switch

kube_pod_annotations{annotation_cluster_autoscaler_kubernetes_io_safe_to_evict=“false”, annotation_cni_projectcalico_org_container_id=“8a505a530b501ad80ce471e86b553257e4ec3541313bc4245233f60a04dd3619”, annotation_cni_projectcalico_org_pod_ip=“10.233.105.3/32”, annotation_cni_projectcalico_org_pod_ips=“10.233.105.3/32”, app_kubernetes_io_component=“metrics”, app_kubernetes_io_instance=“prometheus”, app_kubernetes_io_managed_by=“Helm”, app_kubernetes_io_name=“kube-state-metrics”, app_kubernetes_io_part_of=“kube-state-metrics”, app_kubernetes_io_version=“2.3.0”, exported_namespace=“tekton-pipelines”, helm_sh_chart=“kube-state-metrics-4.4.3”, instance=“10.233.105.11:8080”, job=“kubernetes-service-endpoints”, namespace=“monitor”, node=“node4”, pod=“tekton-pipelines-controller-6f449d874b-mc7nl”, service=“prometheus-kube-state-metrics”, uid=“412f8383-1c5c-4f61-8198-453bdb204911”}

Many labels starting with annotation_ will be added.

After enabling these two switches, the pressure on Prometheus memory, CPU, and storage all increases.

In my test environment, the cluster had 2000 of them, of which only 40 were in Running state. When collecting all of them, Prometheus memory consumption instantly increased by about 400 MB, as shown below:

The Pod’s state does not affect kube-status-metrics’ collection of its metrics.