https://www.chenshaowen.com/en/tags/data/Recent content in Data on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Sun, 09 Mar 2025 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/fluid-using-3fs-as-pvc.htmlSun, 09 Mar 2025 00:00:00 +0000Sun, 09 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-using-3fs-as-pvc.html1. Why Connect 3FS to Fluid 3FS is a distributed storage system open-sourced by DeepSeek. Its exceptionally impressive performance test results have made it a hot topic, and its star count has risen rapidly. The team I work on has also been tracking 3FS technically, looking for suitable application scenarios to get the most value out of our AI hardware infrastructure.微信公众号FluidJuiceFSAIDatahttps://www.chenshaowen.com/en/blog/fluid-juicefs-enterprise-maintenance.htmlSat, 18 Jan 2025 01:00:00 +0000Sat, 18 Jan 2025 01:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-juicefs-enterprise-maintenance.html1. Setting Environment Variables 1 2 export NAMESPACE=xxx export PVC=xxx 2. The Dataset Will Not Become Ready 2.1 Fluid Component Problems 1 kubectl -n fluid-system get pod -o wide | grep -v "Running" There may be cases where something did not start up properly. 2.2 Abnormally Terminated Datasets Abnormal resources can cause Fluid resources to restart repeatedly, and manual intervention is required to delete them.微信公众号FluidJuiceFSKubernetesStorageAIDataOperationshttps://www.chenshaowen.com/en/blog/using-fluid-and-s3fs-to-access-s3-storage-and-performance-testing.htmlThu, 05 Dec 2024 00:00:00 +0000Thu, 05 Dec 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-fluid-and-s3fs-to-access-s3-storage-and-performance-testing.htmlThis post uses Fluid 1.0. In later versions the configuration file paths changed, so adjust according to your actual situation. 1. Building the Image 1.1 fluid_config_init.py 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 #!微信公众号s3FluidJuiceFSAIDatahttps://www.chenshaowen.com/en/blog/using-fluid-to-access-obs-storage-and-performance-testing.htmlTue, 22 Oct 2024 00:00:00 +0000Tue, 22 Oct 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-fluid-to-access-obs-storage-and-performance-testing.html1. Jindo Mounting OBS Set the environment variables 1 2 3 4 export ENDPOINT=obs.cn-north-4.myhuaweicloud.com export BUCKET= export AK= export SK= Create the credentials 1 2 3 4 5 6 7 8 9 10 kubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: myobssecret type: Opaque stringData: fs.obs.accessKeyId: ${AK} fs.微信公众号OBSFluidJuiceFSKubernetesAIDataOperationshttps://www.chenshaowen.com/en/blog/using-fluid-to-access-oss-storage-and-performance-testing.htmlTue, 11 Jun 2024 00:00:00 +0000Tue, 11 Jun 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-fluid-to-access-oss-storage-and-performance-testing.html1. Jindo Directly Accelerating OSS Set the environment variables 1 2 3 4 5 6 7 export ENDPOINT=oss-cn-beijing-internal.aliyuncs.com export BUCKET= export AK= export SK= export NAMESPACE=default export PVC=myoss-jindo export URI=datacenter Create the credentials 1 2 3 4 5 6 7 8 9 10 11 kubectl apply -f - <<EOF apiVersion:微信公众号OSSFluidJuiceFSAIDatahttps://www.chenshaowen.com/en/blog/fluid-mount-s3-and-performance-testing.htmlThu, 16 May 2024 00:00:00 +0000Thu, 16 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-mount-s3-and-performance-testing.html1. Packaging the Fluid Runtime Image Create the fluid_config_init.py script 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 #!/usr/bin/env python import json rawStr = "" with open("/etc/fluid/config.json", "r") as f: rawStr = f.微信公众号FluidJuiceFSKubernetesStorageAIDatahttps://www.chenshaowen.com/en/blog/fluid-using-lustre-runtime-and-performance-testing.htmlThu, 16 May 2024 00:00:00 +0000Thu, 16 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-using-lustre-runtime-and-performance-testing.html1. Analyzing How Fluid Mounts NFS Storage Inspect the Fuse Pod 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 kubectl get pod nfs-demo-fuse-f9wg8 -oyaml apiVersion: v1 kind: Pod metadata: generateName: nfs-demo-fuse- spec: containers: - command: - /usr/local/bin/entrypoint.微信公众号FluidJuiceFSAIDataLustrehttps://www.chenshaowen.com/en/blog/fluid-using-nfs-runtime-and-performance-testing.htmlTue, 14 May 2024 00:00:00 +0000Tue, 14 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-using-nfs-runtime-and-performance-testing.html1. Creating a Dataset 1 2 3 4 5 6 7 8 9 10 kubectl apply -f - <<EOF apiVersion: data.fluid.io/v1alpha1 kind: Dataset metadata: name: nfs-demo spec: mounts: - mountPoint: x.x.x.x:/x-x/ name: nfs-demo EOF 2. Creating a Runtime 1 2 3 4 5 6 7 8 9 10 11 12 13 14 kubectl apply -f - <<EOF apiVersion: data.微信公众号FluidJuiceFSKubernetesStorageAIDatahttps://www.chenshaowen.com/en/blog/sql-code-about-django-orm.htmlFri, 21 Jul 2017 13:21:08 +0000Fri, 21 Jul 2017 13:21:08 +0000https://www.chenshaowen.com/en/blog/sql-code-about-django-orm.html1. Basic Concepts ORM: Object Relational Mapping. Its job is to map between a relational database and objects. No complex SQL statements are needed — operating on data is as simple as operating on objects. QuerySet: the list of objects for a given model. A QuerySet lets you read data微信公众号DjangoPythonDataSQLR&Dhttps://www.chenshaowen.com/en/blog/haystack-full-text-search.htmlFri, 21 Jul 2017 13:00:03 +0000Fri, 21 Jul 2017 13:00:03 +0000https://www.chenshaowen.com/en/blog/haystack-full-text-search.htmlA quick word on the project requirements: the team needed to publish documentation externally. The documents are written in Markdown and need to be published as HTML. At first we used an Nginx + Jekyll solution. As the documentation grew, the document system developed a strong need for search. I微信公众号DjangoPythonDataDesignSearchR&Dhttps://www.chenshaowen.com/en/blog/101-of-django-rest-framework.htmlThu, 22 Jun 2017 23:50:15 +0000Thu, 22 Jun 2017 23:50:15 +0000https://www.chenshaowen.com/en/blog/101-of-django-rest-framework.htmlIn the SaaS development I work on, there is a fairly high demand for development efficiency. From project initiation, prototype design and evaluation, requirements confirmation, frontend design, and backend development through to final acceptance, we complete one iteration in just a few weeks. Guided by agile development, we began rolling微信公众号DataBackendInterfaceDjangoPythonAPIR&Dhttps://www.chenshaowen.com/en/blog/how-to-export-excel-quickly-in-web-development.htmlThu, 18 May 2017 17:51:05 +0000Thu, 18 May 2017 17:51:05 +0000https://www.chenshaowen.com/en/blog/how-to-export-excel-quickly-in-web-development.htmlIn web development you often run into the need to export data. This post mainly covers how to quickly export data and save it as an Excel file. 1. Frontend In web development, formatted data is usually displayed as a table. Below is an employee salary table; we will use微信公众号DjangoFrontendDataDemohttps://www.chenshaowen.com/en/blog/several-methods-of-django-data-migration.htmlTue, 17 Jan 2017 23:29:15 +0000Tue, 17 Jan 2017 23:29:15 +0000https://www.chenshaowen.com/en/blog/several-methods-of-django-data-migration.html1. Background In the course of Web development, migrating data between multiple environments (local, testing, and production) comes up often. This article mainly discusses the possible data migration paths involved in Django development, and seeks feasible methods for them. 2. Scenarios There are four data migration targets in total: the微信公众号DjangoPythonDataMigrationR&D