Using Fluid to Access OSS Storage and Performance Testing

1. Jindo Directly Accelerating OSS

Set the environment variables

1
2
3
4
5
6
7
export ENDPOINT=oss-cn-beijing-internal.aliyuncs.com
export BUCKET=
export AK=
export SK=
export NAMESPACE=default
export PVC=myoss-jindo
export URI=datacenter

Create the credentials

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ${PVC}
  namespace: ${NAMESPACE}
type: Opaque
stringData:
  fs.oss.accessKeyId: ${AK}
  fs.oss.accessKeySecret: ${SK}
EOF

Create the Dataset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: ${PVC}
  namespace: ${NAMESPACE}
spec:
  mounts:
    - mountPoint: oss://${BUCKET}/${URI}/
      options:
        fs.oss.endpoint: ${ENDPOINT}
      name: default
      path: "/"
      encryptOptions:
        - name: fs.oss.accessKeyId
          valueFrom:
            secretKeyRef:
              name: ${PVC}
              key: fs.oss.accessKeyId
        - name: fs.oss.accessKeySecret
          valueFrom:
            secretKeyRef:
              name: ${PVC}
              key: fs.oss.accessKeySecret
  accessModes:
    - ReadWriteMany
EOF

Create the JindoRuntime

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: JindoRuntime
metadata:
  name: ${PVC}
  namespace: ${NAMESPACE}
spec:
  replicas: 1
  fuse:
    image: registry.cn-shanghai.aliyuncs.com/jindofs/jindo-fuse
    imageTag: 6.4.0
    args:
      - -oattr_timeout=60
  tieredstore:
    levels:
      - mediumtype: SSD
        path: /cache
        quota: "40960"
EOF

Create the Pod workload

1
export IMAGE=shaowenchen/demo:ubuntu

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${PVC}-deploy
  namespace: ${NAMESPACE}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ${PVC}-demo
  template:
    metadata:
      labels:
        app: ${PVC}-demo
    spec:
      containers:
        - name: demo
          image: ${IMAGE}
          volumeMounts:
            - mountPath: /data
              name: data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: ${PVC}
EOF

1.1 JuiceFS Performance Test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
juicefs bench --block-size 4096 --big-file-size 1024 --threads 30 ./

BlockSize: 4096 MiB, BigFileSize: 1024 MiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 30
+------------------+---------------+-----------------+
|       ITEM       |     VALUE     |       COST      |
+------------------+---------------+-----------------+
|   Write big file | 1520.22 MiB/s |    20.21 s/file |
|    Read big file | 1595.94 MiB/s |    19.25 s/file |
| Write small file |   8.9 files/s | 3373.29 ms/file |
|  Read small file | 289.1 files/s |  103.79 ms/file |
|        Stat file | 496.8 files/s |   60.39 ms/file |
+------------------+---------------+-----------------+

1.2 DD Performance Test

dd write, 421 MB/s

1
2
3
4
5
6
7
time dd if=/dev/zero of=./dd.txt bs=4M count=2500

10485760000 bytes (10 GB, 9.8 GiB) copied, 24.8855 s, 421 MB/s

real 0m25.047s
user 0m0.004s
sys 0m2.857s

dd first read, 485 MB/s

1
2
3
4
5
6
7
time dd if=./dd.txt of=/dev/null bs=4M count=2500

10485760000 bytes (10 GB, 9.8 GiB) copied, 21.6259 s, 485 MB/s

real 0m21.683s
user 0m0.000s
sys 0m1.451s

dd second read, 533 MB/s

1
2
3
4
5
6
7
time dd if=./dd.txt of=/dev/null bs=4M count=2500

10485760000 bytes (10 GB, 9.8 GiB) copied, 19.6688 s, 533 MB/s

real 0m19.692s
user 0m0.004s
sys 0m1.284s

1.3 Cleaning Up Resources

1
2
3
4
kubectl -n $NAMESPACE delete deployments ${PVC}-deploy
kubectl -n $NAMESPACE delete jindoruntimes ${PVC}
kubectl -n $NAMESPACE delete dataset ${PVC}
kubectl -n $NAMESPACE delete secret ${PVC}

2. JuiceFS Community Edition Connecting to OSS

2.1 Setting the Environment Variables

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
export REDIS_IP=x.x.x.x
export REDIS_PORT=6379
export REDIS_USER=default
export REDIS_PASSWORD=mypassword
export REDIS_DIRECTSERVER=redis://${REDIS_USER}:${REDIS_PASSWORD}@${REDIS_IP}:${REDIS_PORT}/1

export ACCESS_KEY=xxx
export SECRET_KEY=xxx
export BUCKET=xxx
export ENDPOINT=oss-cn-beijing-internal.aliyuncs.com
export BUCKET_ENPOINT=$BUCKET.$ENDPOINT

2.2 Initializing the Filesystem

Install JuiceFS

1
curl -sSL https://d.juicefs.com/install | sh -

Initialize the filesystem

1
2
3
4
5
juicefs format \
        --storage oss \
        --bucket ${BUCKET_ENPOINT}\
        ${REDIS_DIRECTSERVER} \
        oss-direct

3. Mounting JuiceFS Directly on a Host

3.1 JuiceFS Performance Test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
juicefs mount -d --buffer-size 2000 --max-uploads 150 ${REDIS_DIRECTSERVER} ./oss-direct --cache-dir=/data/jfs-oss-direct

+------------------+-------------------+---------------+
|       ITEM       |       VALUE       |      COST     |
+------------------+-------------------+---------------+
|   Write big file |     2348.54 MiB/s |  13.08 s/file |
|    Read big file |     5988.49 MiB/s |   5.13 s/file |
| Write small file |     867.1 files/s | 34.60 ms/file |
|  Read small file |   35705.2 files/s |  0.84 ms/file |
|        Stat file |  103844.2 files/s |  0.29 ms/file |
|   FUSE operation | 534217 operations |    0.91 ms/op |
|      Update meta |   9543 operations |    0.10 ms/op |
|       Put object |  10680 operations |  154.07 ms/op |
|       Get object |   7680 operations |   71.21 ms/op |
|    Delete object |      0 operations |    0.00 ms/op |
| Write into cache |   4314 operations |    1.10 ms/op |
|  Read from cache |   3000 operations |    0.16 ms/op |
+------------------+-------------------+---------------+

3.2 DD Performance Test

dd write, 1.7 GB/s

1
2
3
4
5
6
7
time dd if=/dev/zero of=./dd.txt bs=4M count=2500

10485760000 bytes (10 GB, 9.8 GiB) copied, 5.99897 s, 1.7 GB/s

real 0m6.001s
user 0m0.004s
sys 0m3.112s

dd first read, 369 MB/s

1
2
3
4
5
6
7
time dd if=./dd.txt of=/dev/null bs=4M count=2500

10485760000 bytes (10 GB, 9.8 GiB) copied, 28.4491 s, 369 MB/s

real 0m29.033s
user 0m0.000s
sys 0m3.808s

dd second read, 6.3 GB/s

1
2
3
4
5
6
7
time dd if=./dd.txt of=/dev/null bs=4M count=2500

10485760000 bytes (10 GB, 9.8 GiB) copied, 1.65887 s, 6.3 GB/s

real 0m1.660s
user 0m0.000s
sys 0m1.659s

4. Mounting JuiceFS in a Pod

4.1 Creating the Test Workload

Create the secret

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: juicefs-direct-secret
type: Opaque
stringData:
  metaurl: redis://${REDIS_USER}:${REDIS_PASSWORD}@${REDIS_IP}:6379/1
  access-key: ${ACCESS_KEY}
  secret-key: ${SECRET_KEY}
EOF

Create the Dataset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: juicefs-direct-demo
spec:
  accessModes:
    - ReadWriteMany
  mounts:
    - name: oss-direct
      mountPoint: "juicefs:///"
      options:
        bucket: ${BUCKET_ENPOINT}
        storage: oss
      encryptOptions:
        - name: metaurl
          valueFrom:
            secretKeyRef:
              name: juicefs-direct-secret
              key: metaurl
        - name: access-key
          valueFrom:
            secretKeyRef:
              name: juicefs-direct-secret
              key: access-key
        - name: secret-key
          valueFrom:
            secretKeyRef:
              name: juicefs-direct-secret
              key: secret-key
EOF

Note that the name here needs to stay consistent with the name used in format.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: JuiceFSRuntime
metadata:
  name: juicefs-direct-demo
spec:
  replicas: 1
  tieredstore:
    levels:
      - mediumtype: SSD
        path: /cache
        quota: 40960
EOF

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: juicefs-direct-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: juicefs-direct-demo
  template:
    metadata:
      labels:
        app: juicefs-direct-demo
    spec:
      containers:
        - name: demo
          image: shaowenchen/demo:ubuntu
          volumeMounts:
            - mountPath: /data/jfs
              name: data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: juicefs-direct-demo
EOF

4.2 JuiceFS Performance Test

Enter the Pod and run curl -sSL https://d.juicefs.com/install | sh - to install the JuiceFS client.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
juicefs bench --block-size 4096 --big-file-size 1024 --threads 30 ./

+------------------+-------------------+---------------+
|       ITEM       |       VALUE       |      COST     |
+------------------+-------------------+---------------+
|   Write big file |      754.37 MiB/s |  40.72 s/file |
|    Read big file |     1808.45 MiB/s |  16.99 s/file |
| Write small file |     628.6 files/s | 47.72 ms/file |
|  Read small file |    1129.1 files/s | 26.57 ms/file |
|        Stat file |  120037.9 files/s |  0.25 ms/file |
|   FUSE operation | 536005 operations |    3.39 ms/op |
|      Update meta |   9547 operations |    0.32 ms/op |
|       Put object |  10680 operations |   80.47 ms/op |
|       Get object |  15152 operations |   50.53 ms/op |
|    Delete object |      0 operations |    0.00 ms/op |
| Write into cache |      0 operations |    0.00 ms/op |
|  Read from cache |      0 operations |    0.00 ms/op |
+------------------+-------------------+---------------+

4.3 DD Performance Test

dd write, 794 MB/s

1
2
3
4
5
6
7
time dd if=/dev/zero of=./dd.txt bs=4M count=2500

10485760000 bytes (10 GB, 9.8 GiB) copied, 13.198 s, 794 MB/s

real 0m13.199s
user 0m0.004s
sys 0m2.860s

dd first read, 301 MB/s

1
2
3
4
5
6
7
time dd if=./dd.txt of=/dev/null bs=4M count=2500

10485760000 bytes (10 GB, 9.8 GiB) copied, 34.8118 s, 301 MB/s

real 0m35.162s
user 0m0.004s
sys 0m3.222s

dd second read, 7.0 GB/s

1
2
3
4
5
6
7
time dd if=./dd.txt of=/dev/null bs=4M count=2500

10485760000 bytes (10 GB, 9.8 GiB) copied, 1.48848 s, 7.0 GB/s

real 0m1.490s
user 0m0.000s
sys 0m1.489s

5. Problems Encountered During Use

5.1 JindoRuntime Mounts an OSS Directory but Sees No Data

Symptom: After running for a while, the previously mounted OSS directory no longer shows any data.

Solution: Delete the instance’s jindofs-master-0 several times; after it is rebuilt automatically, it can be recovered hot.

Cause: The jindofs-master-0 logs show no exceptions; the root cause is still under investigation.

Delete the jindofs-master-0 instance in a given namespace

1
2
NS="default"
kubectl -n "$NS" get pod | grep jindofs-master | awk '{print $1}' | xargs kubectl -n "$NS" delete pod

Delete the jindofs-master-0 instance in all namespaces

1
kubectl get pod --all-namespaces | grep jindofs-master | awk '{print $1, $2}' | xargs -n2 sh -c 'kubectl -n $0 delete pod $1'

5.2 jindofs-master Keeps Crashing

Symptom: jindofs-master-0 reports an error

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
4# JfsxMainBase::checkAndSetExitCode(int, bool) at /root/workspace/code/jindocache/jfsx-common/include/JfsxMainBase.hpp:97
 5# JfsxMainBase::runImpl() at /root/workspace/code/jindocache/jfsx-common/src/JfsxMainBase.cpp:57
 6# JfsxMainBase::run() at /root/workspace/code/jindocache/jfsx-common/src/JfsxMainBase.cpp:22
 7# main at /root/workspace/code/jindocache/jfsx-nsmain/main_entry.cpp:37
 8# __libc_start_main at ../csu/libc-start.c:342
 9# 0x00000000005D75BE in /smartdata/sbin/jindocache-server


Error: signal 11 Segmentation fault, PID 1, TID 7f10f9f9f3c0
Stacktrace with Glog:
    @     0x7f10f9fc5611  abort
    @           0x8b6b92  JfsxMainBase::checkAndSetExitCode()
    @           0x429831  _ZN12JfsxMainBase7runImplEv.cold
    @           0x8b9710  JfsxMainBase::run()
    @           0x4f572e  main
    @     0x7f10f9fc709b  __libc_start_main
    @           0x5d75be  (unknown)
    @              (nil)  (unknown)


Stacktrace with Boost:
 0# 0x0000000000FD2B83 in /smartdata/sbin/jindocache-server
 1# 0x00007F10F9FDA970 in /lib/x86_64-linux-gnu/libc.so.6
 2# __GI_abort at /build/glibc-6iIyft/glibc-2.28/stdlib/abort.c:107
 3# JfsxMainBase::checkAndSetExitCode(int, bool) at /root/workspace/code/jindocache/jfsx-common/include/JfsxMainBase.hpp:97
 4# JfsxMainBase::runImpl() at /root/workspace/code/jindocache/jfsx-common/src/JfsxMainBase.cpp:57
 5# JfsxMainBase::run() at /root/workspace/code/jindocache/jfsx-common/src/JfsxMainBase.cpp:22
 6# main at /root/workspace/code/jindocache/jfsx-nsmain/main_entry.cpp:37
 7# __libc_start_main at ../csu/libc-start.c:342
 8# 0x00000000005D75BE in /smartdata/sbin/jindocache-server

Solution: Check the jindoruntime-controller logs. It turns out that some Jindo instances have abnormal data, which prevents the Controller from working properly; the abnormal data needs to be repaired.

Cause: jindoruntime-controller processes Jindo instances serially, so an abnormal Jindo instance means the configuration of subsequent instances never gets updated, and at that point smartdata reports an exception.

5.3 Storage Is Slow to Become Ready

Symptom: After a Dataset is created, it is slow to become ready, configuration updates are slow, and so on.

Solution: Increase the resource limits of jindoruntime-controller. The default CPU limit is 100m; it can be raised to 2000m.

Cause: The CPU used by jindoruntime-controller is being throttled.

5.4 Hot-Fixing the Mount Point

Enter jindo-master and check whether anything is mounted

1
jindo fs -ls jindo:///

If nothing is mounted, try mounting it

1
jindocache -mount / oss://my-bucket/path-dir

After exiting the jindo-master node, you need to restart the Fuse Pod used by the business application for the change to take effect.

5.5 Scripts for Repairing Mounts

List the jindo mount points in a namespace

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
NAMESPACE="default"

datasets=$(kubectl -n "$NAMESPACE" get dataset -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n')

for dataset in $datasets; do
  echo "🔍 Dataset: $dataset"

  master_pods=$(kubectl -n "$NAMESPACE" get pod -l release="$dataset",role=jindofs-master -o name)
  if [[ -z "$master_pods" ]]; then
    echo "❌ 未找到 master pod，跳过 $dataset"
    echo "---------------------------------------"
    continue
  fi

  for pod in $master_pods; do
    pod_name=${pod##*/}
    echo "📦 Master Pod: $pod_name"
    echo "   正在列出挂载路径..."

    output=$(timeout 3s kubectl -n "$NAMESPACE" exec "$pod_name" -- jindo fs -ls jindo:/// 2>/dev/null)

    if [[ -n "$output" ]]; then
      echo "$output" | sed 's/^/   /'
    else
      echo "⚠️  未挂载或无返回"
    fi

    echo "---------------------------------------"
  done
done

Repair the mount points of all abnormal jindo instances in a namespace; the business application’s Pod must be restarted for the change to take effect.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
NAMESPACE="default"

datasets=$(kubectl -n "$NAMESPACE" get dataset -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n')

for dataset in $datasets; do
  echo "处理 Dataset: $dataset"

  oss_path=$(kubectl -n "$NAMESPACE" get dataset "$dataset" -o jsonpath='{.spec.mounts[0].mountPoint}')
  if [[ -z "$oss_path" ]]; then
    echo "未找到 mountPoint，跳过 $dataset"
    continue
  fi
  echo "OSS 路径: $oss_path"

  master_pods=$(kubectl -n "$NAMESPACE" get pod -l release="$dataset",role=jindofs-master -o name)
  if [[ -z "$master_pods" ]]; then
    echo "未找到 master pod，跳过 $dataset"
    continue
  fi

  restart_needed=false

  for pod in $master_pods; do
    pod_name=${pod##*/}
    echo "检查 master pod $pod_name 是否已挂载..."
    ls_output=$(timeout 2s kubectl -n "$NAMESPACE" exec "$pod_name" -- jindo fs -ls jindo:/// 2>/dev/null)

    if [[ -n "$ls_output" ]]; then
      echo "已挂载，跳过"
    else
      echo "执行挂载: jindocache -mount / $oss_path"
      timeout 10s kubectl -n "$NAMESPACE" exec "$pod_name" -- jindocache -umount /
      timeout 10s kubectl -n "$NAMESPACE" exec "$pod_name" -- jindocache -mount / "$oss_path"
      restart_needed=true
    fi
  done

  if [ "$restart_needed" = true ]; then
    echo "挂载完成，重启 fuse pod（release=$dataset, role=jindofs-fuse）"
    fuse_pods=$(kubectl -n "$NAMESPACE" get pod -l release="$dataset",role=jindofs-fuse -o name)
    for fpod in $fuse_pods; do
      fpod_name=${fpod##*/}
      echo "删除 pod: $fpod_name"
      kubectl -n "$NAMESPACE" delete pod "$fpod_name" --force
    done
  else
    echo "未执行挂载操作，跳过 fuse pod 重启"
  fi

  echo "Dataset $dataset 处理完成"
  echo "---------------------------------------"
done

5.6 The Dataset Never Becomes Ready

After an exception occurs and you repair it, the Dataset may still not recover automatically; you can try restarting the node’s kubelet.

1
systemctl restart kubelet

5.7 Jindo Runtime cacheset Exception

The error log is as follows:

1
message: 'execute command [jindocache -refreshCacheSet -path /cacheset.xml] with expectedErr: command terminated with exit code 161 stdout ERROR: code=4001, message=cant''t set cacheset name to default which used by system default cacheset and stderr '

How to fix:

1
kubectl -n $NAMESPACE edit cm ${PVC}-jindofs-config

Remove the default field from the cacheset.xml file.

1
2
3
<cacheset>
  <name>default</name>
</cacheset>

After saving, restart the jindofs-master-0 instance. So far no parameter for configuring cacheset has been found, so it can only be removed manually.

5.8 Pod FailedMount Exception

The related error events

1
2
3
4
Warning  FailedMount       10m (x4 over 24m)     kubelet           Unable to attach or mount volumes: unmounted volumes=[my-pvc-v3-8-1 my-pvc-v3-5-0], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
Warning  FailedMount       8m37s (x4 over 22m)   kubelet           Unable to attach or mount volumes: unmounted volumes=[my-pvc-v3-5-0 my-pvc-v3-8-1], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
Warning  FailedMount       2m14s (x16 over 26m)  kubelet           MountVolume.SetUp failed for volume "ai-prod-my-pvc-v3-8-1" : rpc error: code = Internal desc = timeout waiting for FUSE mount point to be ready
Warning  FailedMount       107s (x3 over 6m20s)  kubelet           Unable to attach or mount volumes: unmounted volumes=[my-pvc-v3-8-1], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition

How to fix

Delete the Fuse Pod for this storage that runs on this node, and it recovers.

6. Summary

The JuiceFS performance test results are as follows

Test scenario	Large file write speed	Large file read speed	Small file write speed	Small file read speed
Jindo-accelerated OSS	1520.22 MiB/s	1595.94 MiB/s	8.9 files/s	289.1 files/s
JuiceFS + OSS on a host	2348.54 MiB/s	5988.49 MiB/s	867.1 files/s	35705.2 files/s
JuiceFS + OSS in a Pod	754.37 MiB/s	1808.45 MiB/s	628.6 files/s	1129.1 files/s

The DD performance test results are as follows

Test scenario	Write speed	First read speed	Second read speed
Jindo-accelerated OSS	421 MB/s	485 MB/s	533 MB/s
JuiceFS + OSS on a host	1.7 GB/s	369 MB/s	6.3 GB/s
JuiceFS + OSS in a Pod	794 MB/s	301 MB/s	7.0 GB/s

Based on the test results above, on Alibaba Cloud you can simply use JindoRuntime to mount OSS as a PVC in a Pod, which is enough to meet model inference requirements.

Using Fluid to directly accelerate object storage for loading models at inference time is a highly recommended approach. It not only saves you from deploying JuiceFS’s metadata storage service, but also enables bidirectional synchronization between the PVC and OSS, which is a great convenience operationally.

7. References

https://github.com/aliyun/alibabacloud-jindodata/blob/latest/docs/user/zh/fluid-jindoruntime/jindo_fluid_ways_to_use.md