1. Testing the Full Bandwidth
- Start the iperf3 server on the target host
1
2
3
4
5
| iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
|
- Test from the client host
1
| iperf3 -c x.x.x.x -p 5201 -t 10
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| Connecting to host x.x.x.x, port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 151 MBytes 1.27 Gbits/sec 3562 164 KBytes
[ 5] 1.00-2.00 sec 134 MBytes 1.12 Gbits/sec 230 164 KBytes
[ 5] 2.00-3.00 sec 124 MBytes 1.04 Gbits/sec 250 213 KBytes
[ 5] 3.00-4.00 sec 122 MBytes 1.03 Gbits/sec 229 195 KBytes
[ 5] 4.00-5.00 sec 122 MBytes 1.03 Gbits/sec 342 198 KBytes
[ 5] 5.00-6.00 sec 125 MBytes 1.05 Gbits/sec 351 212 KBytes
[ 5] 6.00-7.00 sec 124 MBytes 1.04 Gbits/sec 319 220 KBytes
[ 5] 7.00-8.00 sec 121 MBytes 1.02 Gbits/sec 286 220 KBytes
[ 5] 8.00-9.00 sec 125 MBytes 1.05 Gbits/sec 233 191 KBytes
[ 5] 9.00-10.00 sec 122 MBytes 1.03 Gbits/sec 377 192 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.24 GBytes 1.07 Gbits/sec 6179 sender
[ 5] 0.00-10.00 sec 1.24 GBytes 1.06 Gbits/sec receiver
|
This shows that the bandwidth between the two hosts is 1 Gbps.
2. Confirming That Calico Has the Bandwidth Plugin Enabled
- The Bandwidth plugin should be visible in Calico’s ConfigMap configuration
1
| kubectl -n calico-system get cm cni-config -o yaml
|
1
2
3
4
| {
"type": "bandwidth",
"capabilities": {"bandwidth": true}
},
|
- The calico-bandwidth plugin can be seen on the host
1
| cat /etc/cni/net.d/10-calico.conflist
|
1
2
3
4
| {
"type": "bandwidth",
"capabilities": {"bandwidth": true}
},
|
3. Creating a Pod with a Bandwidth Limit
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
| cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-rate-limit
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: test-rate-limit
template:
metadata:
labels:
app: test-rate-limit
annotations:
kubernetes.io/egress-bandwidth: "1M"
kubernetes.io/ingress-bandwidth: "1M"
spec:
containers:
- name: main
image: shaowenchen/demo:iperf3
ports:
- containerPort: 80
EOF
|
Here the M unit is Mbps, so this sets a bandwidth of 1 Mbps.
kubernetes.io/egress-bandwidth is the Pod’s egress bandwidth, that is, the bandwidth at which other applications access this Pod.
kubernetes.io/ingress-bandwidth is the Pod’s ingress bandwidth, that is, the bandwidth at which this Pod accesses other applications.
4. Testing Bandwidth Inside the Pod
- Start the client inside the container
1
| kubectl exec deployment/test-rate-limit -- bash -c "iperf3 -c x.x.x.x -p 5201 -t 10"
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| Connecting to host x.x.x.x, port 5201
[ 5] local 10.244.101.19 port 60566 connected to x.x.x.x port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 8.81 MBytes 73.9 Mbits/sec 0 382 KBytes
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 0 388 KBytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 395 KBytes
[ 5] 3.00-4.00 sec 816 KBytes 6.68 Mbits/sec 0 395 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 395 KBytes
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 395 KBytes
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 0 395 KBytes
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 395 KBytes
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 395 KBytes
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 395 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.61 MBytes 8.06 Mbits/sec 0 sender
[ 5] 0.00-10.11 sec 8.03 MBytes 6.67 Mbits/sec receiver
|
- Set kubernetes.io/egress-bandwidth to 100M
1
| kubectl exec deployment/test-rate-limit -- bash -c "iperf3 -c x.x.x.x -p 5201 -t 10"
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| Connecting to host x.x.x.x, port 5201
[ 5] local 10.244.101.50 port 34504 connected to x.x.x.x port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 127 MBytes 1.06 Gbits/sec 2262 1.37 KBytes
[ 5] 1.00-2.00 sec 124 MBytes 1.04 Gbits/sec 2423 198 KBytes
[ 5] 2.00-3.00 sec 25.0 MBytes 210 Mbits/sec 26 177 KBytes
[ 5] 3.00-4.00 sec 11.2 MBytes 94.4 Mbits/sec 0 177 KBytes
[ 5] 4.00-5.00 sec 11.2 MBytes 94.4 Mbits/sec 0 177 KBytes
[ 5] 5.00-6.00 sec 11.2 MBytes 94.4 Mbits/sec 0 177 KBytes
[ 5] 6.00-7.00 sec 12.5 MBytes 105 Mbits/sec 0 177 KBytes
[ 5] 7.00-8.00 sec 11.2 MBytes 94.4 Mbits/sec 0 177 KBytes
[ 5] 8.00-9.00 sec 11.2 MBytes 94.4 Mbits/sec 0 177 KBytes
[ 5] 9.00-10.00 sec 11.2 MBytes 94.4 Mbits/sec 0 177 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 355 MBytes 298 Mbits/sec 4711 sender
[ 5] 0.00-10.01 sec 352 MBytes 295 Mbits/sec receiver
|
The longer the iperf3 test runs, the more stable the bandwidth value and the closer it gets to the configured limit.
5. Summary
In a cluster, besides CPU and Memory, bandwidth is also a constrained resource within IO. Especially in densely deployed clusters, bandwidth contention frequently increases service latency.
This post has mainly described how to use Calico’s Bandwidth plugin to limit a Pod’s bandwidth, by setting kubernetes.io/egress-bandwidth and kubernetes.io/ingress-bandwidth to limit the Pod’s egress and ingress bandwidth.
It is worth noting that although the Pod’s bandwidth is limited, it does not take effect in real time. When traffic starts, there may be a momentary bandwidth overshoot, but as time goes on the bandwidth gradually stabilizes near the limit value.
During testing I also found that if a Pod uses HostNetwork, the configured bandwidth limit is ineffective. This is because a HostNetwork Pod uses the host’s network resources directly, so it cannot be limited through Calico’s Bandwidth plugin.
