Observing Kubernetes Network Connections with Kindling

I recently had a requirement to collect Kubernetes’ external network access. So I set up and tried out some related projects. This post is mainly about how to install Kindling and configure Grafana to view Kubernetes network connection data.

1. What Is Kindling

Kindling solves the problem of how to observe the network without intruding on the application; its functionality is mainly achieved by exposing kernel events for observation. If the host kernel version is higher than 4.14, you can use the eBPF module; if the host kernel is an older version, it uses Sysdig for the relevant observation.

Here is an architecture diagram:

Kindling currently has two versions, one open-source and one commercial. The open-source version does not collect detailed enough data and can only produce Grafana charts; the commercial version has enhanced functionality, which is introduced on the project’s GitHub homepage, so I will not repeat it here.

This kind of technology — based on eBPF, converting kernel function calls into userspace events and then exposing them to user programs — should have some real-world use cases in the coming years, and will also be quite interesting. So let’s do a simple deployment and try out the open-source version of Kindling.

2. Installing Kindling

The Kindling open-source community does not operate particularly well, and its documentation and materials are not clear enough, so I put together a Yaml file for installation.

2.1 Make Sure the Kernel Version Is Greater Than 4.14

1
2
3
uname -a

Linux node1 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

2.2 Download the Yaml File

1
2
3
git clone https://github.com/shaowenchen/demo

cd kindling/yaml

2.3 Install Kindling

1
kubectl apply -f ./

2.4 Check the Pod Status

1
2
3
4
5
6
7
8
9
kubectl -n kindling get pod

NAME                   READY   STATUS    RESTARTS   AGE
kindling-agent-8xt7c   1/1     Running   0          2d20h
kindling-agent-bvxzc   1/1     Running   0          2d20h
kindling-agent-l9phl   1/1     Running   0          2d20h
kindling-agent-nx5zh   1/1     Running   0          2d20h
kindling-agent-qd9cs   1/1     Running   0          2d20h
kindling-agent-sxglf   1/1     Running   0          2d20h

2.5 The Pod May Stay in CrashLoopBackOff

If the Pod stays in CrashLoopBackOff, this is caused by a mismatch between the kindling-agent image and the current system. You need to recompile the image:

Install kernel headers

On Ubuntu, run:

1
apt-get -y install linux-headers-$(uname -r)

On CentOs, run:

1
yum -y install kernel-devel-$(uname -r)

Compile and generate a new image

1
bash -c "$(curl -fsSL https://k8s-bpf-probes-public.oss-cn-hangzhou.aliyuncs.com/recompile-module.sh)"

Tag the image as the one defined in the Yaml

1
docker tag kindlingproject/kindling-agent:bymyself shaowenchen/kindling-agent:ubuntu-20.04

Just restart the Pod

1
kubeclt -n kindling delete pod kindling-agent-xxx

A trick when replacing the kindling-agent image

You need to change the kindling-agent’s image pull policy to IfNotPresent.

1
imagePullPolicy: IfNotPresent

Note here that if your infrastructure is fairly uniform, with only one operating system and one kernel version, you can tag the image as your own private image, push it to a remote registry, and simply modify the kindling-agent Daemonset image address.

If your infrastructure is not uniform, and one cluster contains multiple kinds of hosts, operating systems, and kernel versions, you can compile them one by one on those special systems. Since a Daemonset can only set one image name, you need to keep the image name of the kindling-agent recompiled on all Kubernetes nodes consistent.

3. Installing the Grafana Plugin and Importing the Dashboard

3.1 Install the topo-plugin Plugin

Since the Grafana I usually use is deployed with Docker, the plugin installation process is slightly more complicated.

Download the plugin

1
2
git clone https://github.com/shaowenchen/demo
cd kindling/dashboard

Copy the plugin into the container

1
docker cp topo-plugin.tar.gz 392fe26ae57f:/var/lib/grafana/plugins/

Here 392fe26ae57f is the container ID that Grafana is running in.

Enter the container and create a directory

1
docker exec -it 392fe26ae57f sh

In the container, create a directory and extract the plugin

1
2
3
4
5
6
7
8
9
cd /var/lib/grafana/plugins/

mkdir kindlingproject-topology-panel

mv topo-plugin.tar.gz kindlingproject-topology-panel/

cd kindlingproject-topology-panel/

tar xvf topo-plugin.tar.gz

Configure the plugin

/etc/grafana/grafana.ini is a read-only file, so you need to copy it outside the container, modify it, and then copy it back to overwrite the original file.

1
docker cp 392fe26ae57f:/etc/grafana/grafana.ini grafana.ini

Edit the grafana.ini file locally and add the following content

1
2
[plugins]
allow_loading_unsigned_plugins = kindlingproject-topology-panel

Copy the modified grafana.ini file back into the container to overwrite the original file

1
docker cp grafana.ini  392fe26ae57f:/etc/grafana/grafana.ini

Restart Grafana

1
docker restart 392fe26ae57f

3.2 Import the Grafana Dashboard

The Grafana version I use is 8.3.1.

The related Dashboard Json files are backed up at https://github.com/shaowenchen/demo/tree/master/kindling/dashboard.

Compared with the Dashboard officially provided by Kindling, the main addition is a DataSource field for switching data sources, which makes it convenient to view monitoring data on different clusters.

4. Viewing the Data Reported by kindling-agent

Here are the screenshots:

On the dashboard you can see some DNS and four-tuple related information, and even the network topology between namespaces and Services. The resource consumption is also acceptable:

1
2
3
4
5
6
7
8
9
kubectl -n kindling top pod

NAME                   CPU(cores)   MEMORY(bytes)
kindling-agent-8xt7c   32m          444Mi
kindling-agent-bvxzc   31m          337Mi
kindling-agent-l9phl   63m          413Mi
kindling-agent-nx5zh   62m          255Mi
kindling-agent-qd9cs   99m          452Mi
kindling-agent-sxglf   34m          701Mi

In some of the charts above, you can find that some data is missing, and some fields show NOT_FOUND_INTERNAL, so the project’s experience is not particularly good.

Through a PromQL statement, I obtained the list of IPs that the cluster accesses externally.

1
count by (dst_ip) (kindling_trace_request_duration_nanoseconds{dst_ip!~"127..*|10..*|172..*"})