1. Create the Dataset
1
2
3
4
5
6
7
8
9
10
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: my-s3
type: Opaque
stringData:
aws.accessKeyId: xxx
aws.secretKey: xxx
EOF
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
name: my-s3
spec:
mounts:
- mountPoint: s3://BUCKET/
name: s3
options:
alluxio.underfs.s3.endpoint: ks3-cn-beijing-internal.ksyun.com
alluxio.underfs.s3.disable.dns.buckets: "false"
encryptOptions:
- name: aws.accessKeyId
valueFrom:
secretKeyRef:
name: my-s3
key: aws.accessKeyId
- name: aws.secretKey
valueFrom:
secretKeyRef:
name: my-s3
key: aws.secretKey
accessModes:
- ReadWriteMany
EOF
|
2. Create the Runtime
1
2
3
4
5
6
7
8
9
10
11
12
| kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: AlluxioRuntime
metadata:
name: my-s3
spec:
tieredstore:
levels:
- mediumtype: MEM
path: /dev/shm
quota: 50Gi
EOF
|
3. Create a Test Pod
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: s3-demo
spec:
containers:
- name: demo
image: shaowenchen/demo:ubuntu
volumeMounts:
- mountPath: /data
name: data
volumes:
- name: data
persistentVolumeClaim:
claimName: my-s3
EOF
|
1
2
3
4
5
6
7
8
9
| time dd if=/dev/zero of=./dd.txt bs=4M count=250
250+0 records in
250+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 57.8283 s, 18.1 MB/s
real 1m14.210s
user 0m0.000s
sys 0m0.363s
|
1
2
3
4
5
| time cp ./dd.txt /dev/null
real 0m13.823s
user 0m0.000s
sys 0m0.386s
|
- Read test with cache, 4424.7 MB/s
1
2
3
4
5
| time cp ./dd.txt /dev/null
real 0m0.226s
user 0m0.004s
sys 0m0.221s
|
5. Cleanup
1
2
3
4
| kubectl delete pod s3-demo
kubectl delete alluxioruntime my-s3
kubectl delete dataset my-s3
kubectl delete secret my-s3
|
6. Summary
Mounting S3 directly as a PVC with Fluid does not perform particularly well; in the tests above the bucket and the cluster were in the same region.
- An already-mounted PVC does not see updates to the S3 bucket. In other words, the PVC is only a Snapshot of the S3 bucket at the moment it was mounted
- Do not create directories inside the PVC; the directories will not be synced to the S3 bucket
- Do not create files inside the PVC; the files will not be synced to the S3 bucket
- Both read and write speeds are very slow
The s3fs + ThinRuntimeProfile approach should be better, but there is no business requirement for it at the moment; also, if you are using OSS, JindoRuntime can accelerate object storage.
