1. top Shows Node Resource Usage Above 100%

1
2
3
4
5
6

kubectl top node

NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
master-1        995m         16%    13760Mi         118%
master-2        827m         13%    10672Mi         92%
master-3        889m         14%    10244Mi         88%

This is because usage is computed against allocatable resources by default, which excludes the portion reserved by Kubelet. In the kubectl source you can see:

1
2
3
4
5
6
7

for _, n := range nodes {
  if !o.ShowCapacity {
    availableResources[n.Name] = n.Status.Allocatable
  } else {
    availableResources[n.Name] = n.Status.Capacity
  }
}

If you need to see the node’s total resource usage, add the --show-capacity flag:

1
2
3
4
5
6

kubectl top node --show-capacity

NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
master-1        1161m        14%    13822Mi         87%
master-2        998m         12%    10640Mi         67%
master-3        877m         10%    10298Mi         65%

In fact, Allocatable and Capacity can be seen directly on the node object:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

kubectl get node master-1  -oyaml

...
status:
  allocatable:
    cpu: "6"
    ephemeral-storage: "284333649859"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 11877928Ki
    pods: "110"
  capacity:
    cpu: "8"
    ephemeral-storage: 308521756Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 16174632Ki
    pods: "110"

The specific reservation amounts can be found in Kubelet’s configuration file /var/lib/kubelet/config.yaml or in the startup flags --system-reserved=cpu=1,memory=2Gi --kube-reserved=cpu=1,memory=2Gi. For details, see https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/reserve-compute-resources/ .

Allocatable = Capacity - Reserved - Evicted Threshold, where Evicted Threshold depends on the resource and is usually a very small value or ratio.

2. top node and Grafana Data Disagree

2.1 free and node_memory_Mem Share the Same Source

Using free to view node resource usage:

1
2
3
4

free -h
              total        used        free      shared  buff/cache   available
Mem:          503Gi        62Gi       243Gi        12Gi       198Gi       426Gi
Swap:            0B          0B          0B

Grafana node resource usage is as follows:

The PromQL used is:

• Total memory, node_memory_MemTotal_bytes{instance=~\"$node\"}
• Used, node_memory_MemTotal_bytes{instance=~\"$node\"} - node_memory_MemAvailable_bytes{instance=~\"$node\"}

Numerically, free and Grafana data are basically consistent.

This is because the node_memory_Mem metrics that Grafana uses, collected by Node Exporter, come from the host’s /proc/meminfo — the same source as free -h output.

2.2 top Uses Metrics Collected by metrics-server

Viewing node resource usage with top

1
2
3

kubectl top node my-node-name
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
my-node-name   4809m        8%     132883Mi        25%

Simulating the top command’s request to metrics-server:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes/my-node-name

{
    "kind": "NodeMetrics",
    "window": "10.292s",
    "usage": {
        "cpu": "5094380203n",
        "memory": "136278224Ki"
    }
}

The memory usage here is about 130 Gi; 130 / 503 = 25.8%, basically consistent with kubectl top node.

2.3 metrics-server’s Data Comes from Kubelet

From the metrics-server source you can see that it is requesting data from Kubelet.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

func (kc *kubeletClient) GetMetrics(ctx context.Context, node *corev1.Node) (*storage.MetricsBatch, error) {
	port := kc.defaultPort
	path := "/metrics/resource"
	nodeStatusPort := int(node.Status.DaemonEndpoints.KubeletEndpoint.Port)
	if kc.useNodeStatusPort && nodeStatusPort != 0 {
		port = nodeStatusPort
	}
	if metricsPath := node.Annotations[AnnotationResourceMetricsPath]; metricsPath != "" {
		path = metricsPath
	}
	addr, err := kc.addrResolver.NodeAddress(node)
	if err != nil {
		return nil, err
	}
	url := url.URL{
		Scheme: kc.scheme,
		Host:   net.JoinHostPort(addr, strconv.Itoa(port)),
		Path:   path,
	}
	return kc.getMetrics(ctx, url.String(), node.Name)
}

Simulating metrics-server’s request to Kubelet

1
2
3
4
5
6
7
8

kubectl get --raw /api/v1/nodes/my-node-name/proxy/metrics/resource |grep node_

# HELP node_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the node in core-seconds
# TYPE node_cpu_usage_seconds_total counter
node_cpu_usage_seconds_total 1.2683530100816046e+08 1721957059813
# HELP node_memory_working_set_bytes [ALPHA] Current working set of the node in bytes
# TYPE node_memory_working_set_bytes gauge
node_memory_working_set_bytes 1.39524251648e+11 1721957059813

As expected, the monitoring data from metrics-server and from the Kubelet API is the same.

2.4 What Is Different About the node_memory_working_set_bytes Metric

• top uses node_memory_working_set_bytes, a metric provided by Kubelet

It includes memory currently in use and active cache, but excludes cache and buffers that can be reclaimed immediately — mainly inactive file cache. Its data comes from /sys/fs/cgroup.

• Grafana uses node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes, metrics provided by Node Exporter

It includes memory currently in use but excludes cache. Its data comes from /proc/meminfo.

As we saw earlier, top shows memory usage of about 130 Gi while Grafana shows about 77 Gi; the 53 Gi difference is cache that cannot be reclaimed immediately. But because the two methods use different data sources, that 53 Gi cannot be analyzed in more detail.

2.5 Kubelet limit Uses container_memory_working_set_bytes

For Pods, the memory usage seen via top and Grafana may be the same, because most Grafana panels plot Pod memory usage with container_memory_working_set_bytes, which matches top’s calculation.

The key question here is: which metric does Kubelet use to evict Pods? The answer is container_memory_working_set_bytes .

container_memory_working_set_bytes better represents a container’s real memory usage.

The figure below illustrates the difference between container_memory_working_set_bytes (about 18GiB) and container_memory_usage_bytes (about 33GiB).

3. Summary

The host kernel version for the data collected in this article is 5.4.0-48-generic. The main points are:

• Because Kubelet reserves resources, top node resource usage may exceed 100%; use --show-capacity to see total resource usage
• The commonly used node resource usage rate (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes ignores active cache, so the rate reads a bit lower than what top node shows. In the example above, Grafana displays 15% usage while top node displays 28%.
• Kubelet uses container_memory_working_set_bytes for Pod eviction, the same memory usage that top pod shows