1. Jindo Mounting OBS
- Set the environment variables
1
2
3
4
| export ENDPOINT=obs.cn-north-4.myhuaweicloud.com
export BUCKET=
export AK=
export SK=
|
1
2
3
4
5
6
7
8
9
10
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: myobssecret
type: Opaque
stringData:
fs.obs.accessKeyId: ${AK}
fs.obs.accessKeySecret: ${SK}
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
name: myobs-jindo
spec:
mounts:
- mountPoint: obs://${BUCKET}/test2/
options:
fs.obs.endpoint: ${ENDPOINT}
name: default
path: "/"
encryptOptions:
- name: fs.obs.accessKeyId
valueFrom:
secretKeyRef:
name: myobssecret
key: fs.obs.accessKeyId
- name: fs.obs.accessKeySecret
valueFrom:
secretKeyRef:
name: myobssecret
key: fs.obs.accessKeySecret
accessModes:
- ReadWriteMany
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: JindoRuntime
metadata:
name: myobs-jindo
spec:
replicas: 2
tieredstore:
levels:
- mediumtype: SSD
path: /cache
quota: 40960
fuse:
args:
- -oro
- -ometrics_port=0
- -okernel_cache
- -oattr_timeout=7200
- -oentry_timeout=7200
- -onegative_timeout=7200
- -opread
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: myobs-jindo
spec:
containers:
- name: demo
image: shaowenchen/demo:ubuntu
volumeMounts:
- mountPath: /data
name: data
volumes:
- name: data
persistentVolumeClaim:
claimName: myobs-jindo
EOF
|
Enter the Pod
1
| kubectl exec -it myobs-jindo -- bash
|
Run
1
| curl -sSL https://d.juicefs.com/install | sh -
|
Install the JuiceFS client
1
2
3
4
5
6
7
8
9
10
11
12
13
| juicefs bench --block-size 4 --big-file-size 1024 /data
Benchmark finished!
BlockSize: 4.0 MiB, BigFileSize: 1.0 GiB, SmallFileSize: 128 KiB, SmallFileCount: 100, NumThreads: 1
+------------------+--------------+----------------+
| ITEM | VALUE | COST |
+------------------+--------------+----------------+
| Write big file | 389.70 MiB/s | 2.63 s/file |
| Read big file | 242.28 MiB/s | 4.23 s/file |
| Write small file | 6.4 files/s | 157.08 ms/file |
| Read small file | 15.5 files/s | 64.39 ms/file |
| Stat file | 34.9 files/s | 28.62 ms/file |
+------------------+--------------+----------------+
|
It is used to distribute models, and the performance is basically sufficient.
3. Cleaning Up Resources
1
2
3
4
| kubectl delete pod myobs-jindo
kubectl delete jindoruntime myobs-jindo
kubectl delete dataset myobs-jindo
kubectl delete secret myobssecret
|
4. References
