How to Warm Up JuiceFS Data

1. About JuiceFS Caching

On a host, warmed cache is placed directly on the host.

In a cluster, there are two levels of cache:

Worker, which provides cache shared at the cluster level
Fuse, which provides cache at the current node level only

2. Warming Data with the JuiceFS Client

Note that warming at the Fuse layer is only effective for the current node; if you need to warm the whole cluster, you must warm at the Worker layer.

Specify a directory

1
juicefs warmup /mnt/jfs/dataset-1

Specify directories in bulk

1
juicefs warmup -f warm.txt

Here warm.txt is a list of directories to warm, one per line

1
2
3
/mnt/jfs/dataset-1
/mnt/jfs/dataset-2
/mnt/jfs/pics

3. Warming Data with Fluid

Warm the specified directories

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
apiVersion: data.fluid.io/v1alpha1
kind: DataLoad
metadata:
  name: juicefs-dataload
spec:
  dataset:
    name: my-dataset
    namespace: default
  options:
    timeout: 120m
  target:
    - path: /jfs/dataset-1
    - path: /jfs/dataset-2
    - path: /jfs/pics

The default timeout is 20 minutes, but because no concurrency -c 200 was configured during warmup, warming was very slow, so it is set to 120 minutes.

If you need to change the concurrency, you can modify the warmup command in the *-loader-data-load-script script in the same namespace, then simply kill the Pod.

Warm the entire dataset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
cat <<EOF > dataload.yaml
apiVersion: data.fluid.io/v1alpha1
kind: DataLoad
metadata:
  name: juicefs-dataload
spec:
  dataset:
    name: juicefs
    namespace: default
EOF

4. Warming Data by Mounting a PVC

Create a Pod that mounts the PVC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
export NAMESPACE=
export PVC_NAME=
export DEMO_IMAGE=
export NODE_NAME=

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: $PVC_NAME
  namespace: $NODE_NAME
spec:
  nodeName: $HOSTNAME
  containers:
    - name: demo
      image: $DEMO_IMAGE
      volumeMounts:
        - mountPath: /data/jfs
          name: data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: $PVC_NAME
EOF

Enter the Pod

1
2
3
kubectl -n $NAMESPACE get pod $PVC_NAME

kubectl -n $NAMESPACE exec -it $PVC_NAME -- bash

Warm the data

Note that if you are using the Enterprise Edition, use the Enterprise Edition JuiceFS client; if you are using the Community Edition, use the Community Edition. Here the Community Edition is used as an example:

1
curl -sSL https://d.juicefs.com/install | sh -

The warmup command is the same as above

1
juicefs warmup /data/jfs/dataset-1