1. Setting Environment Variables

1
2

export NAMESPACE=xxx
export PVC=xxx

2. The Dataset Will Not Become Ready

2.1 Fluid Component Problems

1

kubectl -n fluid-system get pod -o wide | grep -v "Running"

There may be cases where something did not start up properly.

2.2 Abnormally Terminated Datasets

Abnormal resources can cause Fluid resources to restart repeatedly, and manual intervention is required to delete them.

2.3 Checking the Worker \ Fuse Replicas

• worker replicas

1

kubectl -n ${NAMESPACE} get sts -l release=${PVC}

1

kubectl -n ${NAMESPACE} get pod -l release=${PVC},role=juicefs-worker

• fuse replicas

1

kubectl -n kas-job get ds -l release=${PVC}

1

kubectl -n ${NAMESPACE} get pod -l release=${PVC},role=juicefs-fuse

3. Fuse Problems, the PVC Cannot Be Used

3.1 No Fuse Pod Created

The node is missing a label. The following command labels all nodes:

1

kubectl label node fluid.io/f-${NAMESPACE}-${PVC}=true --all

The label format is as follows:

1

fluid.io/f-<namespace>-<pvc>=true

3.2 Problems with the Fuse Mount

1

kubectl -n ${NAMESPACE}  logs -l release=${PVC},role=juicefs-fuse -f --max-log-requests 100

• could not find the requested resource

1

Error from server (NotFound): the server could not find the requested resource ( pods/log ${PVC}-fuse-8xnmd)

• fuse reports an error

Force delete the fuse pod.

3.3 Fuse Reports WithTimeout

The error log looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

goroutine 5248 [select]:
github.com/juicedata/juicefs/pkg/utils.WithTimeout(0xc021778000, 0x4af8e8?)
        /root/go/pkg/mod/github.com/juicedata/juicefs@v1.2.0-dev.0.20240116052722-9bab947db092/pkg/utils/utils.go:112 +0x125
jfs/pkg/chunk.(*cachedStore).put(0xc001424000, {0xc021776020, 0x1f}, 0xc027c6c370)
        /opt/jfs/pkg/chunk/cached_store.go:630 +0x245
jfs/pkg/chunk.(*cachedStore).putWithRetry(0xc001424000, {0xc021776020, 0x1f}, 0x24f10a0?, 0xc025cbb8a0)
        /opt/jfs/pkg/chunk/cached_store.go:688 +0xfd
jfs/pkg/chunk.(*wChunk).syncUpload(0xc025cbb830, {0x3147e80?, 0xc00005c0e0?}, {0xc021776020, 0x1f}, 0xc027c6c370?)
        /opt/jfs/pkg/chunk/cached_store.go:681 +0x295
jfs/pkg/chunk.(*wChunk).upload.func1(0x0)
        /opt/jfs/pkg/chunk/cached_store.go:837 +0x779
created by jfs/pkg/chunk.(*wChunk).upload
        /opt/jfs/pkg/chunk/cached_store.go:749 +0x27b

First check connectivity to the metadata and object storage services, then check whether any residue from a previous Fuse Pod is left on the host.

3.4 Fuse Reports RequestTimeTooSkewed

1
2
3

2025/04/07 23:34:51.861440 juicefs[9] <FATAL>: Storage eos://mybucket/mybucket/ is not configured correctly: Failed to create bucket eos://mybucket/mybucket/: RequestTimeTooSkewed:
        status code: 403, request id: 2ed3a7cc-d222-4980-b890-e68cd10c0088-0067f4d183-57264c0-hubei3-zone1, host id: 57264c0-hubei3-zone1-hubei3, previous error: RequestTimeTooSkewed:
        status code: 403, request id: 38aea478-0fad-4e8a-8701-27dea29b5a0b-0067f4d183-488fe87-hubei3-zone1, host id: 488fe87-hubei3-zone1-hubei3 [format.go:474]

Check whether there is a time offset between the nodes, and install the chronyd service to synchronize the time automatically.

1

systemctl restart chronyd.service && systemctl status chronyd.service

4. Worker Problems, Abnormal Cache

4.1 Worker PartialReady

1. Check the status of the JuiceFSRuntime

1
2
3
4

kubectl -n ${NAMESPACE} get juicefsruntimes.data.fluid.io

NAME    WORKER PHASE   FUSE PHASE   AGE
${PVC}   PartialReady   Ready        32m

2. Check the Worker logs

1
2
3

kubectl -n ${NAMESPACE}  logs -l release=${PVC},role=juicefs-worker -f --max-log-requests 100

Error from server (NotFound): the server could not find the requested resource ( pods/log ${PVC}-worker-53)

3. Delete the abnormal Worker

This kind of Worker may have started normally, but Fluid cannot get its stats status and therefore judges it as not ready.

1

kubectl -n ${NAMESPACE} delete pod ${PVC}-worker-53

4. Remove the label of the abnormal Worker’s node

If the worker selected its node using a nodeSelector, you should remove the label from that node.

4.2 Some Workers Are Abnormal

1

2025/02/13 08:07:10.759238 juicefs[216] <WARNING>: send block chunks/93/67/67065235_8_4194304: peer x.x.x.x:43891: read header: read tcp x.x.x.x:46482->x.x.x.x:43891: i/o timeout [peer.go:338]

In this case, you can delete the Worker and let it restart.

4.3 Insufficient Worker Resources

1

kubectl -n ${NAMESPACE} top pod |grep ai-kas-aigc-jfs-fluid-worker

Check the Worker’s resource usage to see whether it is close to the Limit configuration.

5. Speeds Fluctuate Wildly

Speed fluctuation is mainly a cache problem. First refer to https://juicefs.com/docs/zh/cloud/guide/distributed-cache/#troubleshooting for troubleshooting.

5.1 Workers Across Regions Formed a Distributed Cache

1
2
3

kubectl -n ${NAMESPACE}  logs -l release=${PVC},role=juicefs-worker

2025/01/16 04:12:05.546591 juicefs[189] <INFO>: add peer x.x.102.47:42769 with weight 100 [peer.go:773]

In the Worker logs, add peer lets you find all the cache nodes; check whether there are any nodes that do not match expectations. Nodes in the same cache group should be in the same region.

5.2 No Prewarming on the Worker Nodes

JuiceFS has two levels of cache. If you prewarm on the Fuse, the cache can only be used on the current node; you should prewarm on the Worker nodes so that the cache can be provided to other nodes.

5.3 a Full Disk Causes the Cache to Be Cleaned

JuiceFS Enterprise Edition has a monitoring dashboard; check whether the Block Cache Eviction Rate is abnormal.

When the disk is full or the threshold configured in the JuiceRuntime is reached, the cache is cleaned up, which causes the speed to fluctuate.

In this case, you can clean up cache older than 30d based on the creation time of the cache directories.

1

find /data*/jfs*/*/raw/chunks -maxdepth 3 -type d -mtime +30 -exec rm -rf {} + 2>/dev/null || true

5.4 The low Parameter Is Set Unreasonably

low indicates the minimum space that should be kept on the disk. If it is set too high, JuiceFS will keep cleaning the cache.

In this case, the JuiceFS monitoring shows a low cache hit rate and a high cache cleanup rate.

The solution is to lower the low parameter; the default value is 0.2.

5.5 Some Nodes Are Slow

In the JuiceFS monitoring dashboard you can find Remote Cache Latency to check whether any node is abnormal.

6. Slow Access Speeds

6.1 Inconsistent Cache-Group

When configuring JuiceFS Enterprise Edition with Fluid 1.0.0, if Cache is configured in the DataSet, the Cache-Group of the Fuse and the Worker will be inconsistent.

The JuiceFS monitoring shows a low cache hit rate, only around 30%, whereas normally after prewarming it should be above 80%.

It is best not to configure Cache-Group and let Fluid configure it automatically.

6.2 Pods Take Too Long to Wait When Using the PVC

Set the Fuse cleanPolicy to OnRuntimeDeleted.

cleanPolicy has two possible values:

• OnDemand, clean up when the node no longer needs it
• OnRuntimeDeleted, clean up when the cache is cleaned

If Pods are constantly being created and used, using OnRuntimeDeleted can reduce the Fuse startup time.

7. Operations Recommendations

7.1 Pin the Node Where Each Worker Runs

Pinning the node where a Worker runs prevents cache loss after the Worker restarts.

The specific method is to set the NodeSelector to a specific label, so that the number of labeled nodes equals the number of Worker replicas.

Since the Worker is deployed as a StatefulSet, it starts up one by one in order on each restart, which guarantees that each Worker runs on a fixed node.

8. Installation

• Create the namespace

1

kubectl create ns fluid-system

• Add the Helm repository

1
2

helm repo add fluid https://fluid-cloudnative.github.io/charts
helm repo update

• Install Fluid

1

helm -n fluid-system install fluid --version 1.0.0 --set csi.featureGates='FuseRecovery=true'  --set csi.config.hostNetwork=true  fluid/fluid

Enabling the FuseRecovery feature allows the mount to be recovered automatically when the Fuse restarts. If the feature was not enabled at installation time, you can add the - --feature-gates=FuseRecovery=true startup argument to the CSI configuration.

9. Version Upgrade

Upgrading from 1.0.0 to 1.0.4

• Download Fluid 1.0.4

1

wget https://github.com/fluid-cloudnative/fluid/releases/download/v1.0.4/fluid-1.0.4.tgz

• Check the available parameters

1

tar -zxvf fluid-1.0.4.tgz

1

helm show values ./fluid

• Start the upgrade

1

helm upgrade fluid ./fluid --set crdUpgrade.imagePrefix=registry.cn-beijing.aliyuncs.com/opshub --set crdUpgrade.imageName=fluidcloudnative-fluid-crd-upgrader

Because Fluid pulls images from DockerHub by default, the component-related Deployments and StatefulSets may end up in ImagePullBackOff, which can be solved by changing the image addresses.

During the restart, the cache workers will be restarted and the Fuse needs to be restarted forcefully by hand. Note that restarting the Fuse makes storage unavailable, and the application needs to be restarted to mount it again.

10. The Storage Bucket’s Domain Name Changes

In the Worker and the Fuse, the /root/.juicefs directory holds the complete configuration file, which records the full domain name configuration.

In JuiceFS Community Edition, when we create a Volume, we only enter the same name for the bucket, not the full domain name.

JuiceFS Enterprise Edition automatically selects the internal or public domain name based on the environment, or even the cloud vendor’s endpoint. If the domain name of the storage bucket being accessed changes, it means the network environment has changed.

There are two ways to pin the storage bucket’s domain name:

1. In the JuiceFS Volume configuration, fill in the complete storage bucket domain name, for example http://bucketname.region.aliyuncs.com
2. Specify the endpoint in the client configuration file, for example by mounting the complete /root/.juicefs/my-volume.conf file and indicating the complete storage bucket domain name.