Please enable Javascript to view the contents

在 Kubernetes 下创建后端为 JuiceFS 的 PVC

 ·  ☕ 2 分钟

本篇主要记录创建 JuiceFS PVC 的脚本,方便快速配置。组件部署可以参考 使用 Fluid 和 JuiceFS 在 Kubernetes 管理数据

1. 设置环境变量

  • 桶的配置
1
2
3
4
5
6
export ACCESS_KEY=
export SECRET_KEY=
export BUCKET=
export ENDPOINT=ks3-cn-beijing-internal.ksyun.com
export BUCKET_ENPOINT=$BUCKET.$ENDPOINT
export PROVIDER=ks3
  • Workload 的配置
1
2
3
4
5
export NAMESPACE=
export PVC_NAME=

export NODE_SELECTOR_KEY=
export NODE_SELECTOR_VALUE=
  • 镜像的配置
export JUICEFS_IMAGE=juicedata/juicefs-fuse
export DEMO_IMAGE=shaowenchen/demo-ubuntu
  • 元数据的配置

如果是 Redis 配置

1
2
3
4
export REDIS_PASSWORD=
#ip:port/database
export REDIS_ENDPOINT=
export META_SERVER=redis://:$REDIS_PASSWORD@$REDIS_ENDPOINT

如果是 PostgreSQL 配置

1
2
3
4
5
6
export POSTGRES_USER=admin
export POSTGRES_PASSWORD=
export POSTGRES_IP=
export POSTGRES_PORT=5432
export POSTGRES_DB=
export META_SERVER=postgres://$POSTGRES_USER:$POSTGRES_PASSWORD@$POSTGRES_IP:$POSTGRES_PORT/$POSTGRES_DB?sslmode=disable
1
psql -U $POSTGRES_USER -h $POSTGRES_IP -p $POSTGRES_PORT -d postgres -c "CREATE DATABASE $POSTGRES_DB;"

2. 初始化 JuiceFS

1
2
3
4
5
6
juicefs format \
    --trash-days=1 \
    --storage $PROVIDER \
    --bucket $BUCKET_ENPOINT \
    $META_SERVER \
    $BUCKET

trash-days 为 0 表示禁用回收站。

3. 创建 Secret

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: $PVC_NAME
  namespace: $NAMESPACE
type: Opaque
stringData:
  metaurl: $META_SERVER
  access-key: $ACCESS_KEY
  secret-key: $SECRET_KEY
EOF

4. 创建 Dataset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
  name: $PVC_NAME
  namespace: $NAMESPACE
spec:
  accessModes:
    - ReadWriteMany
  mounts:
    - name: $BUCKET
      mountPoint: "juicefs:///"
      options:
        bucket: $BUCKET_ENPOINT
        storage: $PROVIDER
      encryptOptions:
        - name: metaurl
          valueFrom:
            secretKeyRef:
              name: $PVC_NAME
              key: metaurl
        - name: access-key
          valueFrom:
            secretKeyRef:
              name: $PVC_NAME
              key: access-key
        - name: secret-key
          valueFrom:
            secretKeyRef:
              name: $PVC_NAME
              key: secret-key
EOF

5. 创建 JuiceFSRuntime

注意修改 nodeSelector,因为 worker 会占用大量资源。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
kubectl apply -f - <<EOF
apiVersion: data.fluid.io/v1alpha1
kind: JuiceFSRuntime
metadata:
  name: $PVC_NAME
  namespace: $NAMESPACE
spec:
  replicas: 1
  juicefsVersion:
    image: $JUICEFS_IMAGE
    imageTag: ce-v1.1.0
  fuse:
    image: $JUICEFS_IMAGE
    imageTag: ce-v1.1.0
    cleanPolicy: OnDemand
    options:
      "cache-group": "$PVC_NAME"
  worker:
    nodeSelector:
      $NODE_SELECTOR_KEY: "$NODE_SELECTOR_VALUE"
    resources:
      limits:
        cpu: 15
        memory: 100Gi
      requests:
        cpu: 1
        memory: 10Gi
    options:
      "cache-group": "$PVC_NAME"
  tieredstore:
    levels:
      - mediumtype: SSD
        path: /data/jfs/cache
        quota: 40960   # 40GiB
        low: "0.1"
EOF

对于同一个高速网络下的节点,才能用同一个 cache-group。

6. 创建测试 Pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: $PVC_NAME
  namespace: $NAMESPACE
spec:
  containers:
    - name: demo
      image: $DEMO_IMAGE
      volumeMounts:
        - mountPath: /data/jfs
          name: data
  nodeSelector:
    $NODE_SELECTOR_KEY: "$NODE_SELECTOR_VALUE"
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: $PVC_NAME
EOF

7. 性能测试

1
kubectl -n $NAMESPACE exec -it $PVC_NAME -- bash
1
cd /data/jfs
1
curl -sSL https://d.juicefs.com/install | sh -
1
juicefs bench ./

微信公众号
作者
微信公众号