https://www.chenshaowen.com/en/tags/juicefs/Recent content in JuiceFS on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Sun, 09 Mar 2025 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/fluid-using-3fs-as-pvc.htmlSun, 09 Mar 2025 00:00:00 +0000Sun, 09 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-using-3fs-as-pvc.html1. Why Connect 3FS to Fluid 3FS is a distributed storage system open-sourced by DeepSeek. Its exceptionally impressive performance test results have made it a hot topic, and its star count has risen rapidly. The team I work on has also been tracking 3FS technically, looking for suitable application scenarios to get the most value out of our AI hardware infrastructure.微信公众号FluidJuiceFSAIDatahttps://www.chenshaowen.com/en/blog/fluid-juicefs-enterprise-maintenance.htmlSat, 18 Jan 2025 01:00:00 +0000Sat, 18 Jan 2025 01:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-juicefs-enterprise-maintenance.html1. Setting Environment Variables 1 2 export NAMESPACE=xxx export PVC=xxx 2. The Dataset Will Not Become Ready 2.1 Fluid Component Problems 1 kubectl -n fluid-system get pod -o wide | grep -v "Running" There may be cases where something did not start up properly. 2.2 Abnormally Terminated Datasets Abnormal resources can cause Fluid resources to restart repeatedly, and manual intervention is required to delete them.微信公众号FluidJuiceFSKubernetesStorageAIDataOperationshttps://www.chenshaowen.com/en/blog/using-fluid-and-s3fs-to-access-s3-storage-and-performance-testing.htmlThu, 05 Dec 2024 00:00:00 +0000Thu, 05 Dec 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-fluid-and-s3fs-to-access-s3-storage-and-performance-testing.htmlThis post uses Fluid 1.0. In later versions the configuration file paths changed, so adjust according to your actual situation. 1. Building the Image 1.1 fluid_config_init.py 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 #!微信公众号s3FluidJuiceFSAIDatahttps://www.chenshaowen.com/en/blog/using-fluid-to-access-obs-storage-and-performance-testing.htmlTue, 22 Oct 2024 00:00:00 +0000Tue, 22 Oct 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-fluid-to-access-obs-storage-and-performance-testing.html1. Jindo Mounting OBS Set the environment variables 1 2 3 4 export ENDPOINT=obs.cn-north-4.myhuaweicloud.com export BUCKET= export AK= export SK= Create the credentials 1 2 3 4 5 6 7 8 9 10 kubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: myobssecret type: Opaque stringData: fs.obs.accessKeyId: ${AK} fs.微信公众号OBSFluidJuiceFSKubernetesAIDataOperationshttps://www.chenshaowen.com/en/blog/using-fluid-to-access-oss-storage-and-performance-testing.htmlTue, 11 Jun 2024 00:00:00 +0000Tue, 11 Jun 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-fluid-to-access-oss-storage-and-performance-testing.html1. Jindo Directly Accelerating OSS Set the environment variables 1 2 3 4 5 6 7 export ENDPOINT=oss-cn-beijing-internal.aliyuncs.com export BUCKET= export AK= export SK= export NAMESPACE=default export PVC=myoss-jindo export URI=datacenter Create the credentials 1 2 3 4 5 6 7 8 9 10 11 kubectl apply -f - <<EOF apiVersion:微信公众号OSSFluidJuiceFSAIDatahttps://www.chenshaowen.com/en/blog/store-elasticsearch-data-in-juicefs.htmlWed, 22 May 2024 00:00:00 +0000Wed, 22 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/store-elasticsearch-data-in-juicefs.html1. Storage Options There are three storage approaches: Isolate by directory and share a single JuiceFS Elasticsearch nodes share one JuiceFS instance, mounting different Elasticsearch nodes through subdirectories. /0/ corresponds to node Node-0 /1/ corresponds to node Node-1 /2/ corresponds to node Node-2 The main advantage of this approach is that it is easy to scale and simple to configure.微信公众号KubernetesJuiceFSElasticsearchStorageOperationshttps://www.chenshaowen.com/en/blog/fluid-mount-s3-and-performance-testing.htmlThu, 16 May 2024 00:00:00 +0000Thu, 16 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-mount-s3-and-performance-testing.html1. Packaging the Fluid Runtime Image Create the fluid_config_init.py script 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 #!/usr/bin/env python import json rawStr = "" with open("/etc/fluid/config.json", "r") as f: rawStr = f.微信公众号FluidJuiceFSKubernetesStorageAIDatahttps://www.chenshaowen.com/en/blog/fluid-using-lustre-runtime-and-performance-testing.htmlThu, 16 May 2024 00:00:00 +0000Thu, 16 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-using-lustre-runtime-and-performance-testing.html1. Analyzing How Fluid Mounts NFS Storage Inspect the Fuse Pod 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 kubectl get pod nfs-demo-fuse-f9wg8 -oyaml apiVersion: v1 kind: Pod metadata: generateName: nfs-demo-fuse- spec: containers: - command: - /usr/local/bin/entrypoint.微信公众号FluidJuiceFSAIDataLustrehttps://www.chenshaowen.com/en/blog/fluid-using-nfs-runtime-and-performance-testing.htmlTue, 14 May 2024 00:00:00 +0000Tue, 14 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-using-nfs-runtime-and-performance-testing.html1. Creating a Dataset 1 2 3 4 5 6 7 8 9 10 kubectl apply -f - <<EOF apiVersion: data.fluid.io/v1alpha1 kind: Dataset metadata: name: nfs-demo spec: mounts: - mountPoint: x.x.x.x:/x-x/ name: nfs-demo EOF 2. Creating a Runtime 1 2 3 4 5 6 7 8 9 10 11 12 13 14 kubectl apply -f - <<EOF apiVersion: data.微信公众号FluidJuiceFSKubernetesStorageAIDatahttps://www.chenshaowen.com/en/blog/how-to-quickly-create-juicefs-enterprise-pvc-in-kubernetes.htmlFri, 08 Mar 2024 00:00:00 +0000Fri, 08 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-quickly-create-juicefs-enterprise-pvc-in-kubernetes.htmlThis post mainly records the script for creating an enterprise-edition JuiceFS PVC, to make configuration quick. 1. Set Environment Variables 1.1 Basic Configuration 1 2 3 export NAMESPACE= export PVC_NAME= export JUICEFS_VOLUME= 1.2 JuiceFS Credentials 1 2 3 export TOKEN= export ACCESS_KEY= export SECRET_KEY= 1.3 JuiceFS Service Configuration 1 2 export BASE_URL=http://x.微信公众号AIStorageJuiceFSKubernetesFluidOperationsPVCEnterprise Editionhttps://www.chenshaowen.com/en/blog/how-to-quickly-create-juicefs-pvc-in-kubernetes.htmlThu, 07 Mar 2024 00:00:00 +0000Thu, 07 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-quickly-create-juicefs-pvc-in-kubernetes.htmlThis post mainly records the script for creating a community-edition JuiceFS PVC, to make configuration quick. For component deployment, see Managing Data in Kubernetes with Fluid and JuiceFS. 1. Set Environment Variables Bucket configuration 1 2 3 4 5 6 export ACCESS_KEY= export SECRET_KEY= export BUCKET= export ENDPOINT=ks3-cn-beijing-internal.ksyun.com export BUCKET_ENPOINT=$BUCKET.微信公众号AIStorageJuiceFSKubernetesFluidOperationsPVChttps://www.chenshaowen.com/en/blog/managing-data-in-kubernetes-using-fluid-and-juicefs.htmlSat, 27 Jan 2024 00:00:00 +0000Sat, 27 Jan 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/managing-data-in-kubernetes-using-fluid-and-juicefs.html1. Introduction to Fluid Below is the Fluid architecture diagram from https://github.com/fluid-cloudnative/fluid: Fluid abstracts two concepts: Dataset, a collection of data, the abstraction from the user’s point of view Runtime, the abstraction of the real services such as data storage and acceleration Fluid mainly solves the problems of using traditional caching systems on Kubernetes:微信公众号KubernetesFluidJuiceFSStorageData ManagementAIOperationshttps://www.chenshaowen.com/en/blog/performance-testing-and-comparison-of-juicefs-ce-ee-and-dragonfly.htmlWed, 24 Jan 2024 00:00:00 +0000Wed, 24 Jan 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/performance-testing-and-comparison-of-juicefs-ce-ee-and-dragonfly.htmlNote that --block-size 4096 in this post means 4GB; using --block-size 4 would be more reasonable and would give better write performance. 1. Environment Preparation Enter a safe directory 1 mkdir -p /data/test && cd /data/test All the test tasks are carried out in this directory. Create a separate directory for Redis 1 mkdir redis-data It is recommended to create a new directory, because Redis will change the Owner of the files in the current directory to systemd-coredump.微信公众号JuiceFSDragonflyStoragePerformance TestingKubernetesOperationshttps://www.chenshaowen.com/en/blog/health-check-runs-out-of-root-run-storage.htmlTue, 23 Jan 2024 00:00:00 +0000Tue, 23 Jan 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/health-check-runs-out-of-root-run-storage.htmlIf you use nvidia-container-runtime, this is worth your attention — especially if JuiceFS is also in the picture. 1. An Alert Out of Nowhere, and I Panicked Over the weekend I was studying TensorRT LLM, and while I was at it I installed Dragonfly on the largest production cluster. Then微信公众号KubernetesNVIDIAJuiceFSExperienceTroubleshootingOperations