The DNS Service in Kubernetes

1. About DNS

1.1 What the DNS Service Is For

DNS provides a mapping service from domain names to IPs. For example, you type https://www.chenshaowen.com into a browser to open a page, but the data link communicates over IP and cannot recognize www.chenshaowen.com. This is where a DNS lookup comes in: the input is www.chenshaowen.com and the result is an IP address.

As you can see, DNS provides a mnemonic: we do not have to care about IP addresses and how they change, and only need to remember a string of English characters to find a service.

One common service discovery mechanism is a configuration management center that stores Key/Value pairs, such as Consul, ZooKeeper, and Etcd. The functionality DNS provides can also meet the service discovery needs of a microservice architecture, and Kubernetes takes exactly this approach.

Starting with version 1.11, Kubernetes uses CoreDNS in place of KubeDNS as its built-in DNS service.

1.2 resolv.conf

/etc/resolv.conf is the configuration file of the DNS client, and it has four main parts:

nameserver, the IP address of the DNS server
domain, the suffix of the local domain name
search, the domain suffixes to search
sortlist, which sorts query results in a specific order

The resolver only uses domain and search when it hits a domain name it cannot resolve. For example, when accessing http://abc/index.html, the resolver cannot resolve abc, so it appends the domain or search configuration as a suffix and keeps resolving. When search is configured, domain is ignored.

2. CoreDNS

2.1 Introduction

CoreDNS is a CNCF graduated project. It is a modular, pluggable DNS server built on top of Caddy.

Every plugin follows a specific interface contract. The Corefile uses a DSL to define the DNS service, which makes it easy to enable various plugins and customize the DNS service. Here is an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
example.org:53 {
        errors
        cache 30
        reload
        loop
        forward . 10.0.0.10 {
            prefer_udp
            max_concurrent 100
        }
        prometheus :9153
    }
.:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        rewrite name xxx.xxx.com xxx.svc.cluster.local
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . 10.0.0.10 {
            prefer_udp
            max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
}

This configuration exposes a DNS server listening on port 8000, matching different handling logic per domain name. Each piece of logic loads the specified plugins to handle requests.

Commonly used plugins include:

hosts, configures hosts resolvable across the whole cluster; note that the domain suffix must match search, for example cluster.local, otherwise nodelocaldns cannot report resolutions to coredns
errors, logs errors to stdout
health, provides a health report endpoint
kubernetes, resolves to the IP addresses of Kubernetes cluster services
prometheus, provides a Prometheus metrics endpoint
proxy, forwards queries outside the cluster domain to a designated resolver
cache, enables caching
loop, detects infinite loops and breaks them
reload, automatically reloads the Corefile for hot updates
loadbalance, a DNS load balancer

2.3 Cluster ConfigMap Configuration

View the CoreDNS service:

1
2
3
kubectl -n kube-system get svc coredns

coredns                            ClusterIP   10.233.0.3      <none>        53/UDP,53/TCP,9153/TCP   5h

View the CoreDNS configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
kubectl -n kube-system get cm  coredns -o yaml

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream /etc/resolv.conf
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts {
           127.1.1.1 example.org
           fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf {
          prefer_udp
        }
        cache 30
        loop
        reload
        loadbalance
    }

The default DNS port is 53. Domain names in the cluster.local, in-addr.arpa, and ip6.arpa formats are resolved to the internal IP addresses of Kubernetes.

The fallthrough setting in hosts is very important: unmatched domain names continue on to the next match.

2.4 Operations Recommendations

If you are updating in a production environment, it is recommended to do a canary update at the container level only, rather than letting CoreDNS change nodes.

This is because /etc/resolv.conf may differ between nodes, which would make it impossible to roll the configuration back.

3. NodelocalDNS

3.1 Introduction

To avoid frequent queries to CoreDNS when Pods resolve DNS, NodelocalDNS runs a DNS cache as a DaemonSet on every node to improve cluster performance.

The principle behind NodelocalDNS is to run a Pod in hostNetwork mode and create a network interface bound to the local DNS IP address. When a Pod on a node requests DNS resolution, it is intercepted by NodelocalDNS. NodelocalDNS completes the resolution by reading the cache or requesting DNS from upstream.

3.2 Cluster ConfigMap Configuration

View the NodelocalDNS service

1
2
3
kubectl -n kube-system get ds nodelocaldns

nodelocaldns   1         1         1       1            1           <none>                        5h

View the NodelocalDNS configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
kubectl -n kube-system get cm nodelocaldns -o yaml

apiVersion: v1
data:
  Corefile: |
    cluster.local:53 {
        errors
        cache {
            success 9984 30
            denial 9984 5
        }
        reload
        loop
        bind 169.254.25.10
        forward . 10.233.0.3 {
            force_tcp
        }
        prometheus :9253
        health 169.254.25.10:9254
    }
    in-addr.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.25.10
        forward . 10.233.0.3 {
            force_tcp
        }
        prometheus :9253
    }
    ip6.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.25.10
        forward . 10.233.0.3 {
            force_tcp
        }
        prometheus :9253
    }
    .:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.25.10
        forward . /etc/resolv.conf
        prometheus :9253
    }
...

It also serves DNS on port 53, but NodelocalDNS applies different resolution strategies depending on the domain name. Domain names in the cluster.local, in-addr.arpa, and ip6.arpa formats are resolved by CoreDNS and then cached locally, while everything else is resolved by the node’s DNS and then cached.

3.3 Pointing a Node’s DNS at NodelocalDNS

Edit the parameters

1
vim /var/lib/kubelet/config.yaml

Change the setting that points at CoreDNS

1
2
clusterDNS:
- 10.96.0.10

to point at the NodelocalDNS IP address

1
2
clusterDNS:
- 169.254.25.10

Restart Kubelet

1
systemctl restart kubelet

Pods newly created on this node will then use NodelocalDNS for DNS resolution.

3.4 Operations Recommendations

In production, NodeLocalDNS is sometimes not deployed and CoreDNS is used directly instead.

This is not a good choice, mainly because: if you want to switch DNS services, changing the CoreDNS configuration affects the entire cluster, and you cannot modify only a designated node for a canary test.

For production environments it is strongly recommended to deploy NodeLocalDNS, which guarantees cluster stability.

4. DNS Resolution in a Kubernetes Pod

Create a Pod for testing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
cat > busybox.yaml <<-EOF
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28.4
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF

1
kubectl apply -f busybox.yaml

View the DNS resolution configuration

1
2
3
4
5
kubectl exec busybox cat /etc/resolv.conf

nameserver 169.254.25.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Resolving a normal internal service

1
2
3
4
5
6
7
kubectl exec -ti busybox -- nslookup kubernetes.default

Server:    169.254.25.10
Address 1: 169.254.25.10

Name:      kubernetes.default
Address 1: 10.233.0.1 kubernetes.default.svc.cluster.local

Resolution path: nodelocaldns -> cache -> coredns -> return IP

This involves the logic of search adding a domain suffix, which is not covered here.

Resolving a non-existent service

1
2
3
4
5
6
kubectl exec -ti busybox -- nslookup a.b

Server:    169.254.25.10
Address 1: 169.254.25.10

nslookup: can't resolve 'a.b'

Resolution path: nodelocaldns -> the node’s configured DNS -> not found

Resolving a normal external service

1
2
3
4
5
6
7
kubectl exec -ti busybox -- nslookup www.chenshaowen.com

Server:    169.254.25.10
Address 1: 169.254.25.10

Name:      www.chenshaowen.com
Address 1: 163.181.33.208

Resolution logic: nodelocaldns -> the node’s configured DNS -> return IP

5. ExternalName - CNAME Resolution

An ExternalName Service is a special case of Service. It has no selector and can be used to give an external service an internal alias.

Create an ExternalName Service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
cat > externalname.yaml <<-EOF
apiVersion: v1
kind: Service
metadata:
  name: chenshaowen
  namespace: default
spec:
  type: ExternalName
  externalName: www.chenshaowen.com
EOF

1
kubectl apply -f externalname.yaml

Test access to the internal service

1
2
3
4
5
6
7
kubectl exec -ti busybox -- nslookup chenshaowen.default

Server:    169.254.25.10
Address 1: 169.254.25.10

Name:      chenshaowen.default
Address 1: 58.215.145.110

chenshaowen.default will be mapped to www.chenshaowen.com, which is achieved through a DNS CNAME record.