1. A Strange Requirement

My boss had a strange requirement: use a single kubeconfig file to fetch all kinds of host status information, such as the process list and process states.

My first reaction was that he must not understand containers — how could anyone use them that way, and if you were going to use them that way, why use containers at all? Inside, ten thousand horses were galloping.

Then recently I ran into a command-line tool, and I realized that the clown was me. Let me show you what I found.

2. How Containers Work

A sandbox is a virtual environment; operations performed inside the sandbox have no effect on the outside. Sandboxes are isolated from one another and invisible to one another — they cannot see each other’s existence.

What we usually call a container is a sandbox environment built on Linux’s Cgroups and Namespace technologies.

From the diagram you can see that the boundary between one container and another is controlled by exactly these two technologies, Cgroups and Namespace. Here is a brief description of each:

1. Namespace

Resources under different Namespaces are independent of one another and invisible to one another. Linux gained Mount Namespace support in 2.4.19, UTS and IPS Namespace support in 2.6.19, PID Namespace support in 2.6.24, Network Namespace support in 2.6.29, and User Namespace support in 3.8. Of these, all except User Namespace must be created with root privileges. Meanwhile, 4.6 already added the Cgroup namespace, and RunC (the runtime provided by Docker) has already merged the relevant PR: https://github.com/opencontainers/runc/pull/1916 . Below are the seven kinds of Namespace.

• Mount namespace, which isolates filesystem mount points. Within a namespace, a program’s changes to files affect only its own filesystem and have no effect on other namespaces.

• UTS namespace, which isolates hostname and domain name information. In each namespace, the host and domain information is independent.

• IPC namespace, which isolates the behavior of inter-process communication. Only processes within a single namespace can communicate with one another.

• PID namespace, which isolates the PID space of processes. Process PIDs in different namespaces may repeat without affecting each other. The process with PID 1 is the parent of all other processes, which makes this namespace very meaningful.

• Network namespace, which isolates network resources. Each namespace has its own independent network stack information, so a container runtime appears to be in an independent network.

• User namespace, which isolates users and user groups. The same user can have different roles in different namespaces, which is used to guarantee security.

• Cgroup namespace, which isolates the visibility of Cgroups. In each namespace, there is an independent cgroupns root and cgroup filesystem view.

2. Cgroups

Above we placed a group of processes into a Namespace, isolated externally and sharing resources internally; next we use Cgroups to control their resources. Cgroups provides four capabilities:

• Resource limiting. Set resource consumption limits for a process or process group. For example, exceeding a memory limit causes memory allocation to fail.
• Resource accounting. Track things like CPU time used and memory consumption.
• Task control. Control the state of processes; you can suspend and resume them.
• Priority allocation. Set the priority of processes.

Using the sandbox environment provided by Namespace and Cgroups, plus filesystem technologies, is what supports container technology.

3. A Debugging Tool: nsenter

nsenter is a tool used to enter the Namespace of a specified program and execute commands there. In container scenarios, many containers trim away basic commands such as ip and tcpdump in order to stay lightweight. This makes debugging containers somewhat difficult, and debugging by sharing the Namespace through nsenter solves the problem nicely.

In fact, when RunC creates a container it also calls nsenter, as you can see in the libcontainer code.

1. Installing nsenter

Most Linux operating systems already ship the nsenter command. If yours does not, taking CentOS as an example, run the following command to install the util-linux package:

1

yum install -y util-linux

2. nsenter’s version and parameters

Because different Linux kernels support Namespaces to different degrees, nsenter versions vary. Here we take CentOS 7 as an example:

• Check the system kernel version

1
2
3

uname -a

Linux i-x29a8rdc 3.10.0-1127.10.1.el7.x86_64 #1 SMP Wed Jun 3 14:28:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Check the nsenter version

1
2
3

nsenter -V

nsenter from util-linux 2.23.2

• Check nsenter’s parameters

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

nsenter -h

Usage:
 nsenter [options] <program> [<argument>...]

Run a program with namespaces of other processes.

Options:
 -t, --target <pid>     target process to get namespaces from
 -m, --mount[=<file>]   enter mount namespace
 -u, --uts[=<file>]     enter UTS namespace (hostname etc)
 -i, --ipc[=<file>]     enter System V IPC namespace
 -n, --net[=<file>]     enter network namespace
 -p, --pid[=<file>]     enter pid namespace
 -U, --user[=<file>]    enter user namespace
 -S, --setuid <uid>     set uid in entered namespace
 -G, --setgid <gid>     set gid in entered namespace
     --preserve-credentials do not touch uids or gids
 -r, --root[=<dir>]     set the root directory
 -w, --wd[=<dir>]       set the working directory
 -F, --no-fork          do not fork before exec'ing <program>
 -Z, --follow-context   set SELinux context according to --target PID

 -h, --help     display this help and exit
 -V, --version  output version information and exit

The parameter to note here is -t, which specifies a process from which to obtain the Namespace parameters. The other parameters mainly enable and set things.

Since it is not easy to demonstrate nsenter’s capabilities outside a sandbox environment, let’s experiment further in a container environment.

4. Applying nsenter in a Docker Container Environment

4.1 On the Host, Entering a Container’s Namespace Environment

• Pick a container

1
2
3
4

docker ps

CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                          PORTS                    NAMES
9addecf82c5e        sonarqube:7.9.4-community   "./bin/run.sh"           3 weeks ago         Up 3 weeks                      0.0.0.0:9000->9000/tcp   sonarqube_sonarqube_1

• Get the container’s PID

Every container has a process with PID=1 inside it, just like the init process on a host machine, and it is the parent of all the other processes. But on the host, the container process has a different PID that you can use to manage the container.

1
2
3

docker inspect --format "{{ .State.Pid }}" 9addecf82c5e

3969

• Enter the container’s Namespace environment

Here we take entering the network namespace as an example:

1

nsenter -t 3969  -n /bin/bash

If the host’s default shell exists inside the container, you can omit /bin/bash; otherwise you must explicitly specify a shell that exists in the container.

• Execute the host’s command-line tools to debug the container environment

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
11: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.3/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever

From the output you can see that it displays the network interface address information of the container, yet the ip command comes from the host.

4.2 Inside the Container, Entering the Host’s Namespace Environment

• Create a container in privileged mode that uses the host’s Namespaces

1

docker run --privileged --net=host --ipc=host --pid=host -it --rm docker.io/alpine:3.12 /bin/sh

• Enter the Namespace environment of the PID=1 process

1

nsenter -t 1 -m -u -i -n

• Run commands to operate on the host environment

1
2
3
4
5

docker ps

CONTAINER ID        IMAGE                       COMMAND                  CREATED              STATUS                          PORTS                    NAMES
2cd99b9d7b5a        alpine:3.12                 "/bin/sh"                About a minute ago   Up About a minute                                        trusting_khorana
9addecf82c5e        sonarqube:7.9.4-community   "./bin/run.sh"           3 weeks ago          Up 3 weeks                      0.0.0.0:9000->9000/tcp   sonarqube_sonarqube_1

From the output you can see that the command displays the container information on the host, yet the command is being executed inside the container.

5. Applying nsenter in a Kubernetes Container Environment

This part is similar to the previous section, except that when entering a container you need to go through the Pod to obtain the PID, and when executing commands on the host you need to go through the Pod to create the container.

5.1 From the Host, Entering a Kubernetes Pod to Debug the Container Environment

• Pick a Pod

1
2
3
4

kubectl get pod -o wide

NAME                     READY   STATUS             RESTARTS   AGE     IP             NODE               NOMINATED NODE   READINESS GATES
nginx-6db489d4b7-589bd   1/1     Running            0          11s     10.233.76.91   tf-cd-allinone-0   <none>           <none>

• Get the container ID

1
2
3

kubectl describe pod nginx-6db489d4b7-589bd | grep -A10 "^Containers:" | grep -Eo 'docker://.*$' | head -n 1 | sed 's/docker:\/\/\(.*\)$/\1/'

981c94ef07abfbeca548e9e36cd70a7369d1cf38a50754c2dc4f87fbc27601d1

• Switch to the node the container is on

1

ssh root@tf-cd-allinone-0

• Get the container’s PID

1
2
3

docker inspect --format "{{.State.Pid}}" 981c94ef07abfbeca548e9e36cd70a7369d1cf38a50754c2dc4f87fbc27601d1

6954

• Enter the container’s Namespace

1

nsenter -t 6954 -n

• Execute host commands to debug the container

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
4: eth0@if100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group default
    link/ether 16:a3:44:dc:58:ce brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.233.76.91/32 scope global eth0
       valid_lft forever preferred_lft forever

Note here that a container is bound to a node — in a multi-node environment, after obtaining the container ID you need to switch to the host it is on to operate.

5.2 Inside a Kubernetes Pod, Operating the Host Directly

• Create a file named pod-test.yaml with the following content

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: v1
kind: Pod
metadata:
  name: pod-test
  namespace: default
spec:
  containers:
    - command:
        ["sh", "-c", 'echo "Hello, wwww.chenshaowen.com !" && sleep 3600']
      image: docker.io/alpine:3.12
      name: pod-test
      securityContext:
        privileged: true
  hostIPC: true
  hostNetwork: true
  hostPID: true

• Create the Pod

1

kubectl apply -f pod-test.yaml

• Enter the Pod

1

kubectl exec -it pod-test /bin/sh

• Enter the Namespace of the PID=1 process

1

nsenter -t 1 -m -u -i -n -p

• Run host commands to test

1
2
3
4
5
6

docker ps

CONTAINER ID   IMAGE                                         COMMAND                   CREATED         STATUS         PORTS     NAMES
f6bd778c3172   389fef711851                                  "sh -c 'echo \"Hello,…"   4 minutes ago   Up 4 minutes             k8s_pod-test_pod-test_default_3a496075-419e-477a-b03c-a423677a90be_0
4e197fd98294   kubesphere/pause:3.1                          "/pause"                  4 minutes ago   Up 4 minutes             k8s_POD_pod-test_default_3a496075-419e-477a-b03c-a423677a90be_0
...

Start the container in privileged mode, share Namespaces through the PID=1 process, and execute commands on the host directly.

6. Summary

This post mainly covered how, in a container environment, to escape to the host and execute commands, and how, on the host, to enter a container’s debugging environment. It also gave practical examples for both the Docker and Kubernetes scenarios.

Two things in particular were instructive for me. One is the nsenter command, which deepened my understanding of container technology. The other is that a container started in privileged mode has very broad permissions and must be handled with care; business workloads should run containers in a rootless way wherever possible.

In a Docker Daemon started in privileged mode, you can create a Kubernetes cluster and, through the nsenter command, use nodeSelector to reach any node and then run commands like kubectl/docker/systemctl to carry out destructive activities.

7. References

• https://man7.org/linux/man-pages/man1/nsenter.1.html
• https://docs.docker.com/engine/reference/run/