This page looks best with JavaScript enabled

Distributing Cluster Images with Dragonfly V2

 ·  ☕ 10 min read

1. Introduction to Dragonfly

The Dragonfly documentation at https://d7y.io/zh/docs/ already covers things in detail. Here is just a brief introduction to the main components of V2:

  • Manager, which provides the UI, user management, cluster monitoring, task management, and other features
  • Scheduler, which schedules traffic between Peers and provides preheating and other features
  • Seed Peer, the back-to-source node used to download data from origin sites (Harbor, Docker.io, etc.); it can also act as a Peer node
  • Peer, the terminal node that provides downloaded data

Among these, Manager and Scheduler are separate container images, while Seed Peer and Peer share the same container image.

The image preheating feature Dragonfly supports can be integrated with Harbor, but this article will not cover that. This article mainly introduces some of our practices in the production environment while supporting AI business. It is worth noting that Dragonfly V2 actually builds a P2P distribution network that can distribute not only images but also files, which opens up a lot of possibilities.

2. The Dragonfly Cluster in the IDC

Our AI model inference and training are both based on Kubernetes clusters, and the backend storage uses the enterprise edition of JuiceFS, with several TB of SSD disks mounted on every Node to mount the JuiceFS cache directory.

As a result, every Node in the Kubernetes cluster meets the conditions to act as a Dragonfly Peer node. But when forming the Peer network, we do not want any extra burden, including:

  • NAT traffic across VPCs
  • Data transmission over the public network

Below is the multi-VPC deployment topology of Dragonfly v2 in the IDC:

  • The LB needs a public IP to serve as the Peer access point
  • One VPC corresponds to one Dragonfly Cluster abstraction
  • Although the IDC connects the networks between VPCs, only Peers within a single VPC are allowed to form the network
  • One Peer is deployed on every Node in the cluster

Within a VPC, the following diagram shows the detailed high-availability scheme.

  • The LB only needs an internal IP
  • Use the cloud provider’s MySQL 8.0 and Redis 6 services
  • Two VMs deploy Manager, Scheduler, and Seed Peer
  • Each VM runs a complete Dragonfly cluster, including Manager, Scheduler, and Seed Peer, so it can be used without going through the LB
  • One Peer is deployed on every Node

The P2P distribution network Dragonfly builds should not be coupled too tightly with the PaaS layer, to avoid circular dependencies. Therefore, we use a dual-VM scheme here, sharing data storage to guarantee availability. On the Master nodes of the Kubernetes cluster, we also do not apply any acceleration optimization, to keep the PaaS layer’s control plane simple and independent.

3. Deploying the Dragonfly Control Plane on VMs

Docker needs to be installed in advance, and deployment is done independently on each of the two VMs.

3.1 Installing docker-compose

  • Download docker-compose
1

curl -L https://github.com/docker/compose/releases/download/v2.23.3/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
  • Add the execute permission
1

chmod +x /usr/local/bin/docker-compose
  • Check the version
1

docker-compose -v

3.2 Installing dragonfly

See https://d7y.io/zh/docs/getting-started/quick-start/docker-compose/

  • Download the docker-compose deployment files
1
2
3
4

cd /data
wget https://github.com/dragonflyoss/Dragonfly2/archive/refs/tags/v2.1.28.tar.gz
tar -zxvf v2.1.28.tar.gz
cp -r Dragonfly2-2.1.28/deploy/docker-compose ./
  • Clean up unneeded files
1

rm -rf *2.1.28*
  • Generate the default configuration files
1

cd docker-compose

Since the default release package has no configuration files, we first generate one and then modify it.

1
2

export IP=VM_IP
./run.sh

Terminate execution immediately, then continue modifying the configuration files.

  • Pin the image version
1

sed -i 's/latest/v2.1.28/g' docker-compose.yaml
  • Modify the storage account and other configuration

Modify the Redis and MySQL addresses and passwords

1

vim config/manager.yaml

Modify the Redis password

1

vim config/scheduler.yaml

These two configuration files contain some other configuration items that can be modified according to the actual situation. For example, point the manager’s addr at the current host’s service, output logs to the console, enable Metrics, and so on.

  • Modify seed-peer’s cache directory
1

vim docker-compose.yaml

1
2
3

volumes:
  - ./cache:/var/cache/dragonfly
  - ./data:/var/lib/dragonfly

If you have disabled seed-peer’s ability to act as a peer node, you can skip this step, and the VM’s disk space does not need to be very large either.

  • Start the services
1

docker-compose up -d
  • Check the services
1
2
3
4
5
6

docker-compose ps

NAME        IMAGE                            COMMAND                  SERVICE     CREATED        STATUS                  PORTS
manager     dragonflyoss/manager:v2.1.28     "/opt/dragonfly/bin/…"   manager     14 hours ago   Up 14 hours (healthy)   0.0.0.0:8080->8080/tcp, 0.0.0.0:65003->65003/tcp
scheduler   dragonflyoss/scheduler:v2.1.28   "/opt/dragonfly/bin/…"   scheduler   14 hours ago   Up 14 hours (healthy)   0.0.0.0:8002->8002/tcp
seed-peer   dragonflyoss/dfdaemon:v2.1.28    "/opt/dragonfly/bin/…"   seed-peer   14 hours ago   Up 14 hours (healthy)   65001/tcp, 0.0.0.0:65006-65008->65006-65008/tcp
  • Open the management page and take a look

Visit http://${VM_IP}:8080 to see the Dragonfly management interface. If the machine has no public IP, you can use socat for port forwarding. Pick a machine with a public IP and run the following command to forward port 30000 to port 8080:

1
2

export IP=VM_IP
socat TCP-LISTEN:30000,fork TCP:$IP:8080

Once both VMs are deployed, you can see a cluster like the one below in the Dashboard, with two Schedulers and two Seed Peers. As shown below:

4. Deploying Peer Nodes in the Cluster

Nodes running Peer need access to ports 8002, 65001, 65003, and 65006-65008 on the two VMs.

  • Create the namespace
1

kubectl create ns dragonfly-system
  • Create the configuration file

Here you need to fill the LB’s IP address into the configuration file before the Peer can join the Dragonfly cluster.

1

export MANAGER_IP=LB_IP

There are many parameters that can be modified according to the actual situation. A default configuration file is provided here.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

kubectl apply -f - <<EOF
apiVersion: v1
data:
  dfget.yaml: |
    aliveTime: 0s
    gcInterval: 1m0s
    keepStorage: false
    workHome: /usr/local/dragonfly
    logDir: /var/log/dragonfly
    cacheDir: /var/cache/dragonfly
    pluginDir: /usr/local/dragonfly/plugins
    dataDir: /var/lib/dragonfly
    console: true
    health:
      path: /server/ping
      tcpListen:
        port: 40901
    verbose: true
    pprof-port: 18066
    metrics: ":8000"
    jaeger: ""
    scheduler:
      manager:
        enable: true
        netAddrs:
          - type: tcp
            addr: $MANAGER_IP:65003
        refreshInterval: 10m
      netAddrs:
      scheduleTimeout: 30s
      disableAutoBackSource: false
      seedPeer:
        clusterID: 1
        enable: false
        type: super
    host:
      idc: ""
      location: ""
    download:
      calculateDigest: true
      downloadGRPC:
        security:
          insecure: true
          tlsVerify: true
        unixListen:
          socket: ""
      peerGRPC:
        security:
          insecure: true
        tcpListen:
          port: 65000
      perPeerRateLimit: 5120Mi
      prefetch: false
      totalRateLimit: 10240Mi
    upload:
      rateLimit: 10240Mi
      security:
        insecure: true
        tlsVerify: false
      tcpListen:
        port: 65002
    objectStorage:
      enable: false
      filter: Expires&Signature&ns
      maxReplicas: 3
      security:
        insecure: true
        tlsVerify: true
      tcpListen:
        port: 65004
    storage:
      diskGCThreshold: 1000Gi
      multiplex: true
      strategy: io.d7y.storage.v2.simple
      taskExpireTime: 72h
    proxy:
      defaultFilter: Expires&Signature&ns
      defaultTag:
      tcpListen:
        port: 65001
      security:
        insecure: true
        tlsVerify: false
      registryMirror:
        dynamic: true
        insecure: false
        url: https://index.docker.io
      proxies:
        - regx: blobs/sha256.*
        - regx: s3.*amazonaws.com.*
        - regx: oss.*aliyuncs.com.*
        - regx: obs.*myhuaweicloud.com.*
        - regx: ks3.*ksyun.com.*
    security:
      autoIssueCert: false
      caCert: ""
      certSpec:
        dnsNames: null
        ipAddresses: null
        validityPeriod: 4320h
      tlsPolicy: prefer
      tlsVerify: false
    network:
      enableIPv6: false
    announcer:
      schedulerInterval: 30s
kind: ConfigMap
metadata:
  labels:
    app: dragonfly
  name: dragonfly-dfdaemon
  namespace: dragonfly-system
EOF
  • Create the DaemonSet

We extracted the DaemonSet file from the official Helm Chart. Note that the cache directory the Peer uses is the /data/dfget directory on the host. It is best to clean up the /data/dfget directory on the host in advance to avoid permission issues; there is also no need to create it beforehand, as the DaemonSet will create it automatically.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: dragonfly
  name: dragonfly-dfdaemon
  namespace: dragonfly-system
spec:
  selector:
    matchLabels:
      app: dragonfly
  template:
    metadata:
      labels:
        app: dragonfly
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - image: dragonflyoss/dfdaemon:v2.1.28
        livenessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - -addr=:65000
        name: dfdaemon
        ports:
        - containerPort: 65001
          protocol: TCP
        - containerPort: 40901
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - /bin/grpc_health_probe
            - -addr=:65000
          failureThreshold: 3
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
        volumeMounts:
        - mountPath: /etc/dragonfly
          name: config
        - mountPath: /var/cache/dragonfly
          name: dfgetcache
        - mountPath: /var/lib/dragonfly
          name: dfgetdata
      hostNetwork: true
      hostPID: true
      tolerations:
      - effect: NoSchedule
        operator: Exists
      - effect: NoExecute
        operator: Exists
      volumes:
      - configMap:
          defaultMode: 420
          name: dragonfly-dfdaemon
        name: config
      - hostPath:
          path: /data/dfget/cache
          type: DirectoryOrCreate
        name: dfgetcache
      - hostPath:
          path: /data/dfget/data
          type: DirectoryOrCreate
        name: dfgetdata
EOF
  • Check the workload
1
2
3
4
5
6
7
8

kubectl -n dragonfly-system get pod

NAME                       READY   STATUS    RESTARTS   AGE
dragonfly-dfdaemon-79qkw   1/1     Running   0          14h
dragonfly-dfdaemon-8hhzb   1/1     Running   3          14h
dragonfly-dfdaemon-nnfc5   1/1     Running   0          14h
dragonfly-dfdaemon-w7lff   1/1     Running   0          14h
dragonfly-dfdaemon-wrmzw   1/1     Running   0          14h

5. Deploying Peer Nodes on VMs

  • Create the directory
1

mkdir -p /data/dfget && cd /data/dfget
  • Set the IP
1

wget https://raw.githubusercontent.com/shaowenchen/demo/master/nydus/dfget.template.yaml -O dfget.yaml

1
2

export MANAGER_IP=LB_IP
sed -i "s/__MANAGER_IP__/$MANAGER_IP/g" dfget.yaml
  • Start the Peer
1
2
3
4
5
6

nerdctl run -d --security-opt apparmor=unconfined --security-opt seccomp=unconfined --name=peer --restart=always \
            -p 65000:65000 -p 65001:65001 -p 65002:65002 \
            -v $(pwd)/data:/var/lib/dragonfly \
            -v $(pwd)/cache:/var/cache/dragonfly \
            -v $(pwd)/dfget.yaml:/etc/dragonfly/dfget.yaml:ro \
            dragonflyoss/dfdaemon:v2.1.28

6. Node Configuration

6.1 Docker

Docker’s Mirror approach can only accelerate images from Docker.io, so we use the Proxy approach here, proxying all of Dockerd’s traffic. The difference between Proxy and Mirror is that if Mirror goes down, Dockerd pulls from the origin, whereas if Proxy goes down, Dockerd’s pull simply fails.

  • Add the proxy
1

mkdir -p /etc/systemd/system/docker.service.d

1
2
3
4
5

cat > /etc/systemd/system/docker.service.d/http-proxy.conf <<EOF
[Service]
Environment="HTTP_PROXY=http://127.0.0.1:65001"
Environment="HTTPS_PROXY=http://127.0.0.1:65001"
EOF
  • Restart Docker
1
2

systemctl daemon-reload
systemctl restart docker

Note that if "live-restore": true is not configured in /etc/docker/daemon.json, all containers will be restarted.

  • Check the environment variables
1
2
3

systemctl show --property=Environment docker

Environment=HTTP_PROXY=http://127.0.0.1:65001 HTTPS_PROXY=http://127.0.0.1:65001
  • Image pull test
1

docker pull nginx

At this point, Dockerd’s traffic goes through the Dragonfly Peer node.

6.2 Containerd

See https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration

The config_path = "/etc/containerd/certs.d" item under [plugins."io.containerd.grpc.v1.cri".registry] in /etc/containerd/config.toml provides a mirror-like way of configuring things.

  • Configure Docker.io
1

mkdir -p /etc/containerd/certs.d/docker.io

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

cat > /etc/containerd/certs.d/docker.io/hosts.toml <<EOF
server = "https://docker.io"

[host."http://127.0.0.1:65001"]
  capabilities = ["pull", "resolve"]
  [host."http://127.0.0.1:65001".header]
    X-Dragonfly-Registry = ["https://registry-1.docker.io"]
  [host."https://registry-1.docker.io"]
    capabilities = ["pull", "resolve"]
EOF
  • Configure other, private image registries

The configuration for other image registries can be generated by a script, for example:

1
2

wget https://raw.githubusercontent.com/dragonflyoss/Dragonfly2/main/hack/gen-containerd-hosts.sh
bash gen-containerd-hosts.sh ghcr.io

The reason we did not use the script to generate the docker.io configuration is that in the generated configuration file, X-Dragonfly-Registry is https://docker.io rather than https://registry-1.docker.io.

Using X-Dragonfly-Registry = ["https://docker.io"] produces the following error:

1

unknow type: text/html

The mirror added above takes effect immediately without restarting Containerd.

  • Image pull test
1

nerdctl pull nginx

At this point, you can see the image data cached by the Peer node under the local /data/dfget/data directory.

7. Integrating Nydus

If Nydus has already been configured, it can actually be configured easily here.

  • Add a mirror for Nydusd
1

vim /etc/nydus/nydusd-config.fusedev.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "mirrors": [
          {
            "host": "http://127.0.0.1:65001",
            "auth_through": false,
            "headers": {
              "X-Dragonfly-Registry": "https://index.docker.io"
            },
            "ping_url": "http://127.0.0.1:40901/server/ping"
          }
        ]
      }
    }
  }
}
  • Restart Nydusd
1

systemctl restart nydus-snapshotter
  • Image pull test
1

nerdctl pull shaowenchen/demo:ubuntu:latest-nydus

8. Summary

This article records part of the process of testing and deploying Dragonfly V2 in the production environment this week. The main content includes:

  • The deployment topology of the Dragonfly cluster in the IDC
  • The deployment of Peer nodes on clusters and VMs
  • The integration of Docker, Containerd, and Nydus

As for the shortcomings: there is no metric monitoring. When doing benchmarks, we found that data transmission between Peers within an AZ and across AZs is both limited. If you want to build a high-performance P2P distribution network, the network between Peer and Peer, and between Peer and Seed Peer, is an important factor to consider.

微信公众号
WRITTEN BY
微信公众号

See Also