Fluid Using Lustre Runtime and Performance Testing

1. Analyzing How Fluid Mounts NFS Storage

Inspect the Fuse Pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
kubectl get pod nfs-demo-fuse-f9wg8 -oyaml
apiVersion: v1
kind: Pod
metadata:
  generateName: nfs-demo-fuse-
spec:
  containers:
  - command:
    - /usr/local/bin/entrypoint.sh
    env:
    - name: FLUID_RUNTIME_TYPE
      value: thin
    - name: FLUID_RUNTIME_NS
      value: default
    - name: FLUID_RUNTIME_NAME
      value: nfs-demo
    - name: MOUNT_POINT
      value: /runtime-mnt/thin/default/nfs-demo/thin-fuse
    - name: MOUNT_OPTIONS
      value: ro
    image: fluidcloudnative/nfs:v0.1
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - sh
          - -c
          - umount /runtime-mnt/thin/default/nfs-demo/thin-fuse
    name: thin-fuse
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /etc/fluid/config.json
      name: thin-conf
      readOnly: true
      subPath: config.json
    - mountPath: /etc/fluid/runtime.json
      name: runtime
      readOnly: true
      subPath: runtime.json
  hostNetwork: true

After it starts, Fuse mounts the storage directory onto the Node; before it stops, it unmounts the storage directory.

Inspect the configuration files injected by Fluid

1
2
cat  /etc/fluid/config.json
{"mounts":[{"mountPoint":"x.x.x.x:/x-x","name":"nfs-demo"}],"targetPath":"/runtime-mnt/thin/default/nfs-demo/thin-fuse","accessModes":["ReadOnlyMany"]}

1
2
cat  /etc/fluid/runtime.json
{"workers":[],"fuses":[]}

Fluid injects the mountPoint configuration from the Dataset into the Fuse by mounting it as a Json file.

Inspect the Fuse startup script

1
2
3
4
5
6
7
8
9
cat /usr/local/bin/entrypoint.sh
#!/usr/bin/env bash
set +x

python3 /fluid_config_init.py

chmod u+x /mount-nfs.sh

bash /mount-nfs.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
cat fluid_config_init.py
#!/usr/bin/env python

import json

rawStr = ""
with open("/etc/fluid/config.json", "r") as f:
    rawStr = f.readlines()

rawStr = rawStr[0]

script = """
#!/bin/sh
set -ex
MNT_FROM=$mountPoint
MNT_TO=$targetPath


trap "umount ${MNT_TO}" SIGTERM
mkdir -p ${MNT_TO}
mount -t nfs ${MNT_FROM} ${MNT_TO}
sleep inf
"""

obj = json.loads(rawStr)

with open("mount-nfs.sh", "w") as f:
    f.write("mountPoint=\"%s\"\n" % obj['mounts'][0]['mountPoint'])
    f.write("targetPath=\"%s\"\n" % obj['targetPath'])
    f.write(script)

When the Fuse Pod starts, it parses the config.json file, generates the mount-nfs.sh script, and executes it.

2. Packaging the Fluid Lustre Runtime Image

From the analysis above, we can see that for this kind of mount-based file storage service, all you need is to package a corresponding Fuse image to hook it into Fluid for management.

Create the fluid_config_init.py script

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#!/usr/bin/env python

import json

rawStr = ""
with open("/etc/fluid/config.json", "r") as f:
    rawStr = f.readlines()

rawStr = rawStr[0]

script = """
#!/bin/sh
set -ex
MNT_FROM=$mountPoint
MNT_TO=$targetPath

trap "umount ${MNT_TO}" SIGTERM
mkdir -p ${MNT_TO}
mount -t lustre -o relatime,flock ${MNT_FROM} ${MNT_TO}
sleep inf
"""

obj = json.loads(rawStr)

with open("mount-lustre.sh", "w") as f:
    f.write('mountPoint="%s"\n' % obj["mounts"][0]["mountPoint"])
    f.write('targetPath="%s"\n' % obj["targetPath"])
    f.write(script)

Only the mount command needs adjusting.

Create the startup script entrypoint.sh

1
2
3
4
5
6
#!/usr/bin/env bash
set +x

python /fluid_config_init.py
chmod u+x /mount-lustre.sh
bash /mount-lustre.sh

Create a Dockerfile to package the image

1
2
3
4
5
FROM amazon/aws-fsx-csi-driver:8e204f0ab565dd116bc39699391b6d642a3ae900
COPY ./fluid_config_init.py /
COPY ./entrypoint.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT []

Build the image and push it

1
docker build -t shaowenchen/demo:fluid-lustre . --push

3. Connecting Lustre to Fluid

Create a Dataset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: lustre-demo
spec:
  mounts:
  - mountPoint: fs-x.fsx.us-west-2.amazonaws.com@tcp:/x
    name: lustre-demo
EOF

Note the mountPoint here: if you need to mount the subdirectory subdir, create it in advance. In production, multiple PVCs may share a single Lustre backend.

The format for mounting a subdirectory is: fs-x.fsx.us-west-2.amazonaws.com@tcp:/x/subdir

Create a Runtime

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: ThinRuntimeProfile
metadata:
  name: lustre
spec:
  fileSystemType: lustre
  fuse:
    image: shaowenchen/demo:fluid-lustre
    imageTag: latest
    imagePullPolicy: Always
    command:
      - "/usr/local/bin/entrypoint.sh"
EOF

1
2
3
4
5
6
7
8
kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: ThinRuntime
metadata:
  name: lustre-demo
spec:
  profileName: lustre
EOF

Create a Pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: lustre-demo
spec:
  containers:
    - name: lustre-demo
      image: shaowenchen/demo:ubuntu
      volumeMounts:
        - mountPath: /data
          name: lustre-demo
  volumes:
    - name: lustre-demo
      persistentVolumeClaim:
        claimName: lustre-demo
  tolerations:
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"
EOF

4. Performance Testing

The figure below shows the FSx for Lustre specification we provisioned on AWS.

4.1 Sequential Read Test with a Direct Mount on the Host

Install lustre-client

Ubuntu 20.04

1
2
3
4
wget -O - https://fsx-lustre-client-repo-public-keys.s3.amazonaws.com/fsx-ubuntu-public-key.asc | gpg --dearmor | sudo tee /usr/share/keyrings/fsx-ubuntu-public-key.gpg >/dev/null
bash -c 'echo "deb [signed-by=/usr/share/keyrings/fsx-ubuntu-public-key.gpg] https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu focal main" > /etc/apt/sources.list.d/fsxlustreclientrepo.list && apt-get update'

apt install -y linux-aws lustre-client-modules-$(uname -r)

Ubuntu 24.04

1
2
3
4
wget -O - https://fsx-lustre-client-repo-public-keys.s3.amazonaws.com/fsx-ubuntu-public-key.asc | gpg --dearmor | sudo tee /usr/share/keyrings/fsx-ubuntu-public-key.gpg >/dev/null
bash -c 'echo "deb [signed-by=/usr/share/keyrings/fsx-ubuntu-public-key.gpg] https://fsx-lustre-client-repo.s3.amazonaws.com/ubuntu noble main" > /etc/apt/sources.list.d/fsxlustreclientrepo.list && apt-get update'

apt install -y linux-aws lustre-client-modules-$(uname -r)

Note that you need to run sudo reboot to restart the machine.

Reference documentation: https://docs.aws.amazon.com/zh_cn/fsx/latest/LustreGuide/install-lustre-client.html

Run the test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
fio -direct=1 -iodepth=32 -rw=read -ioengine=libaio -bs=4m -size=10g -numjobs=1 -runtime=1000 -group_reporting -filename=testfile --allow_mounted_write=1 -name=Sequ_Read_Testing

Sequ_Read_Testing: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [R(1)][96.7%][r=368MiB/s][r=92 IOPS][eta 00m:01s]
Sequ_Read_Testing: (groupid=0, jobs=1): err= 0: pid=74694: Thu May 16 18:40:31 2024
  read: IOPS=86, BW=347MiB/s (364MB/s)(10.0GiB/29479msec)
    slat (msec): min=7, max=260, avg=11.51, stdev=14.19
    clat (usec): min=17, max=2646.5k, avg=344495.25, stdev=192586.04
     lat (msec): min=11, max=2655, avg=356.01, stdev=197.77
    clat percentiles (msec):
     |  1.00th=[  284],  5.00th=[  305], 10.00th=[  309], 20.00th=[  313],
     | 30.00th=[  326], 40.00th=[  330], 50.00th=[  334], 60.00th=[  334],
     | 70.00th=[  338], 80.00th=[  338], 90.00th=[  342], 95.00th=[  347],
     | 99.00th=[  435], 99.50th=[ 2601], 99.90th=[ 2635], 99.95th=[ 2635],
     | 99.99th=[ 2635]
   bw (  KiB/s): min=303104, max=417792, per=100.00%, avg=384223.08, stdev=21823.92, samples=53
   iops        : min=   74, max=  102, avg=93.77, stdev= 5.35, samples=53
  lat (usec)   : 20=0.04%
  lat (msec)   : 20=0.04%, 50=0.12%, 100=0.16%, 250=0.55%, 500=98.16%
  lat (msec)   : 750=0.08%, 1000=0.04%, 2000=0.16%, >=2000=0.66%
  cpu          : usr=0.04%, sys=6.67%, ctx=2614, majf=0, minf=32779
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=347MiB/s (364MB/s), 347MiB/s-347MiB/s (364MB/s-364MB/s), io=10.0GiB (10.7GB), run=29479-29479msec

4.2 Sequential Read Test Inside a Pod

Enter the Pod

1
2
3
kubectl exec -it lustre-demo bash

cd /data

Run the test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
fio -direct=1 -iodepth=32 -rw=read -ioengine=libaio -bs=4m -size=10g -numjobs=1 -runtime=1000 -group_reporting -filename=testfile --allow_mounted_write=1 -name=Sequ_Read_Testing

Sequ_Read_TestingA: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=364MiB/s][r=91 IOPS][eta 00m:00s]
Sequ_Read_TestingA: (groupid=0, jobs=1): err= 0: pid=563: Thu May 16 18:11:28 2024
  read: IOPS=78, BW=315MiB/s (330MB/s)(10.0GiB/32490msec)
    slat (msec): min=7, max=261, avg=12.69, stdev=21.49
    clat (usec): min=3, max=5356.2k, avg=357490.74, stdev=319869.00
     lat (msec): min=10, max=5386, avg=370.18, stdev=334.81
    clat percentiles (msec):
     |  1.00th=[  288],  5.00th=[  309], 10.00th=[  313], 20.00th=[  317],
     | 30.00th=[  326], 40.00th=[  334], 50.00th=[  334], 60.00th=[  338],
     | 70.00th=[  342], 80.00th=[  347], 90.00th=[  347], 95.00th=[  351],
     | 99.00th=[  355], 99.50th=[ 3171], 99.90th=[ 5336], 99.95th=[ 5336],
     | 99.99th=[ 5336]
   bw (  KiB/s): min=98304, max=417792, per=100.00%, avg=376955.83, stdev=41482.14, samples=54
   iops        : min=   24, max=  102, avg=92.00, stdev=10.13, samples=54
  lat (usec)   : 4=0.04%
  lat (msec)   : 20=0.04%, 50=0.12%, 100=0.16%, 250=0.51%, 500=98.24%
  lat (msec)   : 750=0.04%, 1000=0.04%, 2000=0.16%, >=2000=0.66%
  cpu          : usr=0.05%, sys=6.02%, ctx=2757, majf=0, minf=32780
  IO depths    : 1=0.1%, 2=0.1%, 4=0.2%, 8=0.3%, 16=0.6%, 32=98.8%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=2560,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=315MiB/s (330MB/s), 315MiB/s-315MiB/s (330MB/s-330MB/s), io=10.0GiB (10.7GB), run=32490-32490msec

In Fluid, the PVC performance of a ThinRuntime loses very little compared with a direct mount on the host. Note here that the blocksize and size heavily affect the test results. If you only read 1g of data, sequential read performance can reach 500+ MB/s; if the blocksize is 128k, sequential read performance drops to only 100+ MB/s. Therefore, adjustments must be made according to the use case in order to get an accurate assessment.

5. Summary

Recently, domestic model inference services needed to be deployed overseas, and we selected AWS FSx for Lustre as the storage backend. But to keep the way the business layer uses storage consistent, we needed to hook Lustre into Fluid.

After a domestic model is uploaded to S3, it is automatically synced to Lustre.

Early versions of Fluid already supported Lustre, but the Fluid community did not provide detailed documentation or a Demo example, so this post mainly records the practice of connecting Lustre using Fluid’s ThinRuntime.

Since we only use it to store inference models, and model data is usually large files, we only tested sequential read speed for performance. Under the specification we chose, the speed inside the PVC can reach 300+ MB/s.