DeepSeek 3FS Operations Guide

Notes on some DeepSeek 3FS operations, continually updated.

1. Basic Concepts and Cautions

Chain

A Chain is made up of several Targets, and each Target is a replica of the storage. When all commits are ready, all Targets in a Chain are consistent.

Targets on the same Chain cannot be on the same node.

A Chain is the unit of storage space: writing a file assigns it to a Chain, and reading a file reads from a Chain.

Target

A Target is a data replica, and Targets are distributed across a group of nodes.

A Target has three Roles: HEAD, MIDDLE, and TAIL. HEAD is the newest data, TAIL is the oldest data, MIDDLE syncs data from HEAD, and TAIL syncs data from MIDDLE.

After a Target is removed, the disk space is released as well.

ChainTable

A ChainTable records a set of Chain IDs and represents the Chains the storage system can use. A Chain is not used to store data until it has been added to a ChainTable.

Parameter constraints

1
2
3
number of chains >= number of disks * targets per disk / replication factor
number of chains >= stripe size
number of disks >= replication factor

Committing configuration

The configuration of MGMTD, META, STORAGE, and FUSE nodes must all be committed before they work properly.

Handle Storage node data with care

When starting a Storage node that already holds data, the Targets in the directory are re-registered into the cluster. Those Targets are then in the TAIL state and will sync data from MIDDLE, which easily leads to data corruption.

It is best not to start a Storage node with data; let 3FS rebuild the data instead.

Go slowly

State synchronization between 3FS components takes some time, so proceed slowly and wait for state synchronization to finish.

Commands are idempotent

A command can be run repeatedly with no harm. When state synchronization lags, you can re-run the command.

2. How to Set the Striping Parameter

When initializing a cluster there is an init-cluster operation, which has a very important parameter strip size, i.e. the stripe size.

Striping splits data into several data blocks distributed across different Chains to improve parallel read/write performance. The stripe size is the number of data blocks the data is split into.

Too large a stripe size increases the complexity of metadata management, and data reconstruction and recovery take longer. Generally, 16 is enough.

If there are too few disks, the striping parameter may be limited. It is best not to run init-cluster first; instead, after generating chainTable and before uploading, run:

1
cat output/generated_chains.csv

1
2
3
4
ChainId,TargetId,TargetId,TargetId
900100001,100100100101,100100200101,100100400101
900100002,100100200102,100100300101,100100500101
...

Check how many chain-id entries there are; the striping parameter cannot exceed the number of Chains.

3. Reset and Initialize the Cluster

Enter fdbcli and clear the metadata

1
2
writemode on
clearrange "" \xFF

Afterwards the nodes go offline, and the ConfigVersion of all nodes is reset to 0.

Restart all Storage nodes

All Storage nodes need to be restarted, and the mounted storage directories cleaned up along the way.

1
rm -rf /data1/3fs/*

1
nerdctl restart 3fs_storage

List all nodes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml list-nodes

Id    Type     Status               Hostname      Pid  Tags  LastHeartbeatTime    ConfigVersion  ReleaseVersion
3     MGMTD    PRIMARY_MGMTD        8c05a1fbdf97  16   []    N/A                  0(UPTODATE)    250228-dev-1-999999-8c9883c2
1     MGMTD    HEARTBEAT_CONNECTED  58ca52320c3b  16   []    2025-03-19 02:20:58  0(UPTODATE)    250228-dev-1-999999-8c9883c2
2     MGMTD    HEARTBEAT_CONNECTED  391173a5a978  16   []    2025-03-19 02:20:59  0(UPTODATE)    250228-dev-1-999999-8c9883c2
101   META     HEARTBEAT_CONNECTED  b3fea43900e6  193  []    2025-03-19 02:21:04  0              250228-dev-1-999999-8c9883c2
102   META     HEARTBEAT_CONNECTED  fcebcb6ca5c3  186  []    2025-03-19 02:21:06  0              250228-dev-1-999999-8c9883c2
103   META     HEARTBEAT_CONNECTED  fea159ba0bd6  186  []    2025-03-19 02:21:00  0              250228-dev-1-999999-8c9883c2
1001  STORAGE  HEARTBEAT_CONNECTED  919404e2891f  187  []    2025-03-19 02:21:06  8(UPTODATE)    250228-dev-1-999999-8c9883c2
1002  STORAGE  HEARTBEAT_CONNECTED  e19b012d073c  179  []    2025-03-19 02:21:05  8(UPTODATE)    250228-dev-1-999999-8c9883c2
1003  STORAGE  HEARTBEAT_CONNECTED  fd2423d5e695  179  []    2025-03-19 02:21:05  8(UPTODATE)    250228-dev-1-999999-8c9883c2
1004  STORAGE  HEARTBEAT_CONNECTED  e8b6fbb9a774  178  []    2025-03-19 02:21:07  8(UPTODATE)    250228-dev-1-999999-8c9883c2
1005  STORAGE  HEARTBEAT_CONNECTED  19db0bc40648  172  []    2025-03-19 02:21:08  8(UPTODATE)    250228-dev-1-999999-8c9883c2

At this point the MGMTD and META configurations have not been committed yet; after the cluster initialization operation, the MGMTD and META nodes need to be restarted again.

Compute data placement

1
2
python3 /opt/3fs/deploy/data_placement/src/model/data_placement.py \
   -ql -relax -type CR --num_nodes 5 --replication_factor 3 --min_targets_per_disk 6

1
2025-03-19 10:02:29.157 | SUCCESS  | __main__:run:148 - saved solution to: output/DataPlacementModel-v_5-b_10-r_6-k_3-λ_2-lb_1-ub_1

Generate chainTable

1
2
3
4
5
python3 /opt/3fs/deploy/data_placement/src/setup/gen_chain_table.py \
   --chain_table_type CR --node_id_begin 1001 --node_id_end 1005 \
   --num_disks_per_node 1 --num_targets_per_disk 6 \
   --target_id_prefix 1 --chain_id_prefix 9 \
   --incidence_matrix_path output/DataPlacementModel-v_5-b_10-r_6-k_3-λ_2-lb_1-ub_1/incidence_matrix.pickle

Check the number of Chains

1
cat output/generated_chains.csv

There are 10 Chains in total.

Initialize the cluster

1
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "init-cluster --mgmtd /opt/3fs/etc/mgmtd_main.toml 1 1048576 10"

Restart the MGMTD and META nodes

1
2
nerdctl restart 3fs_mgmtd
nerdctl restart 3fs_meta

Now all nodes are ready.

Create Targets

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < output/create_target_cmd.txt

Create a user

1
/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "user-add --root --admin 0 root"

1
Token              AACJgD5Y8QCi6ce42wDfTDcr(Expired at N/A)

Save the Token; it is needed for all subsequent operations.

Upload the chainTable

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml "upload-chains output/generated_chains.csv"

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml "upload-chain-table --desc stage 1 output/generated_chain_table.csv"

4. Listing the Clients Using the Storage

1
2
3
4
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml list-clients

ClientId                              ClientStart          SessionStart         LastExtend           ConfigVersion  Hostname                      Description                         Tags  ReleaseVersion
a938c253-1030-4138-b618-0e11113a4876  2025-03-14 08:04:26  2025-03-14 08:04:26  2025-03-18 03:39:39  336            927bb7bb3396                  fuse: 927bb7bb3396                  []    250228-dev-1-999999-8c9883c2

5. Rebuilding a Storage Node

Rebuild a storage node in place on the original machine. This is typically used when the storage directory changes, a disk is replaced, and so on.

Delete the storage node’s data

1
rm -rf /data1/3fs/*

Restart the storage node

1
nerdctl restart 3fs_storage

Check the Targets

1
2
3
4
5
6
7
8
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml  list-targets | grep 1001001

100100100101  900100001  TAIL    OFFLINE      OFFLINE     1001    0          1342177280
100100100105  900100008  TAIL    OFFLINE      OFFLINE     1001    0          4027580416
100100100104  900100005  TAIL    OFFLINE      OFFLINE     1001    0          4027580416
100100100106  900100010  TAIL    OFFLINE      OFFLINE     1001    0          4026531840
100100100102  900100003  TAIL    OFFLINE      OFFLINE     1001    0          4026531840
100100100103  900100004  TAIL    OFFLINE      OFFLINE     1001    0          4027580416

Create Targets

1
2
3
4
5
6
7
8
echo "
create-target --node-id 1001 --disk-index 0 --target-id 100100100101 --chain-id 900100001  --use-new-chunk-engine
create-target --node-id 1001 --disk-index 0 --target-id 100100100105 --chain-id 900100008  --use-new-chunk-engine
create-target --node-id 1001 --disk-index 0 --target-id 100100100104 --chain-id 900100005  --use-new-chunk-engine
create-target --node-id 1001 --disk-index 0 --target-id 100100100106 --chain-id 900100010  --use-new-chunk-engine
create-target --node-id 1001 --disk-index 0 --target-id 100100100102 --chain-id 900100003  --use-new-chunk-engine
create-target --node-id 1001 --disk-index 0 --target-id 100100100103 --chain-id 900100004  --use-new-chunk-engine
" > create_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < create_target_cmd.txt

Wait for the Targets on the node to finish rebuilding

1
2
3
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml  list-targets

100100500101  900100002  TAIL    SYNCING      ONLINE      1005    0          1342177280

A Target in SYNCING means data reconstruction is in progress and you need to wait for it to finish.

6. Replacing a Storage Node

Swap in a new storage node, and the node-id changes as well.

Delete the old node’s Targets

1
2
3
4
5
6
7
8
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml list-targets |grep 1004

100100400101  900100001  MIDDLE  SERVING      UPTODATE    1004    0          4026531840
100100400105  900100008  MIDDLE  SERVING      UPTODATE    1004    0          1342177280
100100400104  900100007  MIDDLE  SERVING      UPTODATE    1004    0          4026531840
100100400102  900100005  HEAD    SERVING      UPTODATE    1004    0          4026531840
100100400106  900100009  MIDDLE  SERVING      UPTODATE    1004    0          4026531840
100100400103  900100006  TAIL    SERVING      UPTODATE    1004    0          4026531840

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
echo "
offline-target --node-id 1004 --target-id 100100400101
update-chain --mode remove 900100001 100100400101
remove-target --node-id 1004 --target-id 100100400101

offline-target --node-id 1004 --target-id 100100400105
update-chain --mode remove 900100008 100100400105
remove-target --node-id 1004 --target-id 100100400105

offline-target --node-id 1004 --target-id 100100400104
update-chain --mode remove 900100007 100100400104
remove-target --node-id 1004 --target-id 100100400104

offline-target --node-id 1004 --target-id 100100400102
update-chain --mode remove 900100005 100100400102
remove-target --node-id 1004 --target-id 100100400102

offline-target --node-id 1004 --target-id 100100400106
update-chain --mode remove 900100009 100100400106
remove-target --node-id 1004 --target-id 100100400106

offline-target --node-id 1004 --target-id 100100400103
update-chain --mode remove 900100006 100100400103
remove-target --node-id 1004 --target-id 100100400103
" > remove_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < remove_target_cmd.txt

Delete the offline Targets

1
2
3
4
5

- Stop the old storage node

```bash
nerdctl rm 3fs_storage -f

Its status becomes HEARTBEAT_FAILED, and the Targets are in the OFFLINE state.

Start a new storage node with the same ID

Set the new node’s node-id to 1006. I tried keeping the new node’s node-id the same as before, but it could not rebuild — even with the same hostname it still would not — so there may be some other unique identifier.

Create Targets and add them to the Chain

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
echo "
create-target --node-id 1006 --disk-index 0 --target-id 100100400101 --chain-id 900100001  --use-new-chunk-engine
update-chain --mode add 900100001 100100400101

create-target --node-id 1006 --disk-index 0 --target-id 100100400105 --chain-id 900100008  --use-new-chunk-engine
update-chain --mode add 900100008 100100400105

create-target --node-id 1006 --disk-index 0 --target-id 100100400104 --chain-id 900100007  --use-new-chunk-engine
update-chain --mode add 900100007 100100400104

create-target --node-id 1006 --disk-index 0 --target-id 100100400102 --chain-id 900100005  --use-new-chunk-engine
update-chain --mode add 900100005 100100400102

create-target --node-id 1006 --disk-index 0 --target-id 100100400106 --chain-id 900100009  --use-new-chunk-engine
update-chain --mode add 900100009 100100400106

create-target --node-id 1006 --disk-index 0 --target-id 100100400103 --chain-id 900100006  --use-new-chunk-engine
update-chain --mode add 900100006 100100400103
" > create_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < create_target_cmd.txt

This triggers data reconstruction for the Targets; wait until their status becomes SERVING.

7. Expanding by Adding a Storage Node

This needs to be planned ahead: the total number of Targets should be an integer multiple of the total number of storage disks. Here we expand from 5 nodes to 6 nodes; the total number of Targets is 30, with 6 Targets per node before the expansion and 5 Targets per node after.

Start a new storage node

The node-id is 1006. After the new node joins, each node should have 5 Targets. Below we need to find 5 Targets and assign them to the new node.

Pick out the Targets to migrate

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml list-targets

You can use an LLM to find the Targets to migrate: list 5 rows of data from it, requiring that they come from different Chains and from different nodes respectively, and output the data in the original format.

1
2
3
4
5
100100100101  900100001  HEAD    SERVING      UPTODATE    1001    0          1342177280
100100200102  900100002  HEAD    SERVING      UPTODATE    1002    0          1342177280
100100300105  900100009  HEAD    SERVING      UPTODATE    1003    0          1342177280
100100400105  900100008  TAIL    SERVING      UPTODATE    1004    0          1342177280
100100500106  900100010  TAIL    SERVING      UPTODATE    1005    0          1342177280

Remove these Targets

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
echo "
offline-target --node-id 1001 --target-id 100100100101
update-chain --mode remove 900100001 100100100101
remove-target --node-id 1001 --target-id 100100100101

offline-target --node-id 1002 --target-id 100100200102
update-chain --mode remove 900100002 100100200102
remove-target --node-id 1002 --target-id 100100200102

offline-target --node-id 1003 --target-id 100100300105
update-chain --mode remove 900100009 100100300105
remove-target --node-id 1003 --target-id 100100300105

offline-target --node-id 1004 --target-id 100100400105
update-chain --mode remove 900100008 100100400105
remove-target --node-id 1004 --target-id 100100400105

offline-target --node-id 1005 --target-id 100100500106
update-chain --mode remove 900100010 100100500106
remove-target --node-id 1005 --target-id 100100500106
" > offline_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < offline_target_cmd.txt

Add new Targets and join them to the Chain

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
echo "
create-target --node-id 1006 --disk-index 0 --target-id 100100100101 --chain-id 900100001  --use-new-chunk-engine
update-chain --mode add 900100001 100100100101

create-target --node-id 1006 --disk-index 0 --target-id 100100200102 --chain-id 900100002  --use-new-chunk-engine
update-chain --mode add 900100002 100100200102

create-target --node-id 1006 --disk-index 0 --target-id 100100300105 --chain-id 900100009  --use-new-chunk-engine
update-chain --mode add 900100009 100100300105

create-target --node-id 1006 --disk-index 0 --target-id 100100400105 --chain-id 900100008  --use-new-chunk-engine
update-chain --mode add 900100008 100100400105

create-target --node-id 1006 --disk-index 0 --target-id 100100500106 --chain-id 900100010  --use-new-chunk-engine
update-chain --mode add 900100010 100100500106
" > create_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < create_target_cmd.txt

At this point the new node immediately starts data reconstruction, and you can check the status with list-targets.

8. Taking a Storage Node Offline

Here we plan to take node 1001 offline and migrate it to other nodes.

Filter out the Targets that need rebuilding

1
2
3
4
5
6
7
8
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml list-targets |grep 1001001

100100100101  900100001  MIDDLE  SERVING      UPTODATE    1001    0          5369757696
100100100105  900100008  MIDDLE  SERVING      UPTODATE    1001    0          2684354560
100100100104  900100005  MIDDLE  SERVING      UPTODATE    1001    0          5369757696
100100100106  900100010  MIDDLE  SERVING      UPTODATE    1001    0          2684354560
100100100102  900100003  TAIL    SERVING      UPTODATE    1001    0          5368709120
100100100103  900100004  TAIL    SYNCING      ONLINE      1001    0          3121610752

Remove the old Targets

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
echo "
offline-target --node-id 1001 --target-id 100100100101
update-chain --mode remove 900100001 100100100101
remove-target --node-id 1001 --target-id 100100100101

offline-target --node-id 1001 --target-id 100100100105
update-chain --mode remove 900100008 100100100105
remove-target --node-id 1001 --target-id 100100100105

offline-target --node-id 1001 --target-id 100100100104
update-chain --mode remove 900100005 100100100104
remove-target --node-id 1001 --target-id 100100100104

offline-target --node-id 1001 --target-id 100100100106
update-chain --mode remove 900100010 100100100106
remove-target --node-id 1001 --target-id 100100100106

offline-target --node-id 1001 --target-id 100100100102
update-chain --mode remove 900100003 100100100102
remove-target --node-id 1001 --target-id 100100100102

offline-target --node-id 1001 --target-id 100100100103
update-chain --mode remove 900100004 100100100103
remove-target --node-id 1001 --target-id 100100100103
" > remove_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < remove_target_cmd.txt

Plan the distribution of Targets

Here we again need an LLM to plan the distribution of Targets.

Feed the output of /opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml list-targets together with the prompt migrate all Targets on node 1001 to other nodes, ensuring that Targets belonging to the same Chain are not on the same node, and output the mapping between the migrated Target Chains and the new nodes into the LLM. The resulting distribution:

Target	Chain	Role	New Node
100100100101	900100001	MIDDLE	1003
100100100105	900100008	MIDDLE	1002
100100100104	900100005	MIDDLE	1004
100100100106	900100010	MIDDLE	1004
100100100102	900100003	TAIL	1005
100100100103	900100004	TAIL	1006

Distribute the Targets evenly across the other nodes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
echo "
create-target --node-id 1003 --disk-index 0 --target-id 100100100101 --chain-id 900100001  --use-new-chunk-engine
update-chain --mode add 900100001 100100100101

create-target --node-id 1002 --disk-index 0 --target-id 100100100105 --chain-id 900100008  --use-new-chunk-engine
update-chain --mode add 900100008 100100100105

create-target --node-id 1004 --disk-index 0 --target-id 100100100104 --chain-id 900100005  --use-new-chunk-engine
update-chain --mode add 900100005 100100100104

create-target --node-id 1004 --disk-index 0 --target-id 100100100106 --chain-id 900100010  --use-new-chunk-engine
update-chain --mode add 900100010 100100100106

create-target --node-id 1005 --disk-index 0 --target-id 100100100102 --chain-id 900100003  --use-new-chunk-engine
update-chain --mode add 900100003 100100100102

create-target --node-id 1006 --disk-index 0 --target-id 100100100103 --chain-id 900100004  --use-new-chunk-engine
update-chain --mode add 900100004 100100100103
" > create_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < create_target_cmd.txt

Wait for the Targets’ status to go from SYNCING to SERVING.

Take the node offline

1
nerdctl rm 3fs_storage -f

The Storage node’s status now becomes HEARTBEAT_FAILED.

1
1001  STORAGE  HEARTBEAT_FAILED     919404e2891f  190  []    2025-03-20 03:13:57  7(UPTODATE)    250228-dev-1-999999-8c9883c2

Delete the node from the list

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml unregister-node 1001 STORAGE

A node can only be taken offline when it is in an abnormal state.

9. Adding a Chain

Adding a Chain increases the number of Targets, and placing Targets on new nodes also achieves expansion.

9.1 Adding a New Chain

Dump the Chains

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml dump-chains dump-for-add-nodes

Inspect the Chains

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
vim dump-for-add-nodes.3

ChainId,TargetId,TargetId,TargetId
900100001,100100200101,100100100101,100100400101
900100002,100100300101,100100500101,100100200102
900100008,100100100105,100100300104,100100400105
900100007,100100200106,100100500104,100100400104
900100005,100100100104,100100500103,100100400102
900100010,100100100106,100100300106,100100500106
900100009,100100500105,100100300105,100100400106
900100003,100100100102,100100200103,100100300102
900100004,100100100103,100100200104,100100500102
900100006,100100200105,100100300103,100100400103

When adding a new Chain, make sure the number of Targets on the Chain is consistent.

1
2
990100001,190000000001,190000000002,190000000003
...

You can add many Chains, but using them also means needing more storage nodes, because Targets on the same Chain cannot be on the same node.

Upload the Chains

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml "upload-chains dump-for-add-nodes.3"

Check the Targets

1
2
3
4
5
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml  list-targets

190000000001   990100001  HEAD    SERVING      OFFLINE     N/A     N/A        0
190000000002   990100001  MIDDLE  SERVING      OFFLINE     N/A     N/A        0
190000000003   990100001  TAIL    SERVING      OFFLINE     N/A     N/A        0

Now there are new Targets waiting to be placed on nodes.

9.2 Adding New Targets

Add a new node

Here we take the newly joined node 1009 as an example, assigning it only some of the Targets.

Assign Targets

1
2
3
4
5
echo "
create-target --node-id 1003 --disk-index 0 --target-id 190000000001 --chain-id 990100001  --use-new-chunk-engine
create-target --node-id 1004 --disk-index 0 --target-id 190000000002 --chain-id 990100001  --use-new-chunk-engine
create-target --node-id 1002 --disk-index 0 --target-id 190000000003 --chain-id 990100001  --use-new-chunk-engine
" > create_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < create_target_cmd.txt

Check the Targets

1
2
3
4
5
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml  list-targets |grep 1900

190000000003  990100001  HEAD    SERVING      UPTODATE    1002    0          0
190000000001  990100001  MIDDLE  SERVING      UPTODATE    1003    0          0
190000000002  990100001  TAIL    SYNCING      ONLINE      1004    0          0

At this point the storage still cannot use this Chain; it needs to be added to the ChainTable.

9.3 Adding a New ChainTable

Dump the ChainTable

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml dump-chain-table 1 ./dump-chainTable

Inspect the ChainTable

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
vim dump-chainTable

ChainId
900100001
900100002
900100003
900100004
900100005
900100006
900100007
900100008
900100009
900100010

Add the ChainId of the new Chain

1
990100001

Upload the ChainTable

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml "upload-chain-table --desc stage 1 ./dump-chainTable"

remove Target

1200000000001 920100001 HEAD SERVING UPTODATE 1001 0 0

1
2
3
echo "
create-target --node-id 1001 --disk-index 0 --target-id 1200000000006 --chain-id 920100002  --use-new-chunk-engine
" > remove_chain_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < remove_chain_cmd.txt

10. Targets in the LASTSRV State

LASTSRV

This state is usually an abnormal state caused by a Target being added to storage ahead of time.

1
2
3
120000000004   930100002  HEAD    LASTSRV      OFFLINE     N/A     N/A        0
120000000005   930100002  MIDDLE  OFFLINE      OFFLINE     N/A     N/A        0
120000000006   930100002  TAIL    OFFLINE      OFFLINE     N/A     N/A        0

Assign the Targets in the OFFLINE state

1
2
3
echo "
create-target --node-id 1001 --disk-index 0 --target-id 120000000005 --chain-id 930100002  --use-new-chunk-engine
" > create_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < create_target_cmd.txt

Rotate the Targets on the Chain

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml rotate-lastsrv 930100002

Check the Targets

1
2
3
120000000005   930100002  HEAD    SERVING      ONLINE      1001    0          0
120000000006   930100002  MIDDLE  OFFLINE      OFFLINE     N/A     N/A        0
120000000004   930100002  TAIL    OFFLINE      OFFLINE     N/A     N/A        0

Now the HEAD state in the Chains has become SERVING.

11. Rebuilding a Target

This applies when a Target is in an abnormal state, for example when its size is inconsistent with other Targets.

Check the Target

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml  list-targets |grep 100100200102

1
100100200102  900100002  TAIL    SYNCING      ONLINE      1002    0          3911188480

Delete the Target

1
2
3
4
5
echo "
offline-target --node-id 1002 --target-id 100100200102
update-chain --mode remove 900100002 100100200102
remove-target --node-id 1002 --target-id 100100200102
" > offline_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < offline_target_cmd.txt

Recreate the Target

1
2
3
4
echo "
create-target --node-id 1002 --disk-index 0 --target-id 100100200102 --chain-id 900100002  --use-new-chunk-engine
update-chain --mode add 900100002 100100200102
" > create_target_cmd.txt

1
/opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml < create_target_cmd.txt

12. Disk Space Used by the Service Gradually Increasing

After the service has been running for a while, the disk space it uses gradually increases and needs to be cleaned up.

Finding many parquet files

1
2
3
4
du -h --max-depth=1

32M ./2026-02-28
16M ./2026-03-01

1
2
ls -alh 2026-02-28/myhost-h20-9/StorageEventTrace.myhost-h20-9.2026-02-28-08-01-13.34.parquet 
-rw-r--r-- 1 root root 68K Feb 28 08:01 2026-02-28/myhost-h20-9/StorageEventTrace.myhost-h20-9.2026-02-28-08-01-13.34.parquet

Reading the parquet files

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
cat << 'EOF' > read_parquet_file.py
import pandas as pd

file = "2026-02-28/myhost-h20-9/StorageEventTrace.myhost-h20-9.2026-02-28-08-01-13.34.parquet"
df = pd.read_parquet(file)

pd.set_option('display.max_columns', None)
pd.set_option('display.width', 1000)

print(df.head())
EOF

python3 read_parquet_file.py

1
2
dateReq_payload_key_vChainId_chainVer        updateReq_payload_key_chunkId                          updateReq_payload_rdmabuf  updateReq_payload_updateVer  updateReq_payload_updateType  updateReq_payload_checksum_type  updateReq_payload_checksum_value updateReq_payload_inlinebuf  updateReq_options_isSyncing  updateReq_options_fromClient  updateReq_options_commitChainVer           updateReq_tag_clientId_uuid updateReq_tag_clientId_hostname  updateReq_tag_requestId  updateReq_tag_channel_id  updateReq_tag_channel_seqnum  updateReq_retryCount  updateReq_userInfo_uid  updateReq_userInfo_gid updateReq_userInfo_groups  updateReq_userInfo_token  updateReq_featureFlags  updateReq_debugFlags_injectRandomServerError  updateReq_debugFlags_injectRandomClientError  \
0           1772265673  myhost-h20-9   ds3fs-h   10009  101000900105                   1048576                   1048576                     16777216                               900100007                                       60  00000000-0000009A-5ACD0000-00000001  {"addr":94591273574400,"size":1048576,"rkeys":...                            2                             1                                1                        1102467336     std::vector<uint8_t>(0)                        False                         False                                 0  b9b6870c-bbb1-40b3-ab43-6b75b20ec845       bj6-e-ai-kas-node-h200-01                514799878                         4                     166861963                     0                       0                       0                        []  AAAvR39B8QC+IPTY2wD24X+I                       0                                         False                                         False   

Inside is a detailed time record of writes to the file system, which can be used for troubleshooting and auditing.

Event trace configuration

Event tracing can be configured in the startup configuration.

1
2
3
4
5
6
[server.storage.event_trace_log]
dump_interval = '30s'
enabled = true
max_num_writers = 1
max_row_group_length = 100000
trace_file_dir = '.'

以下内容需付费阅读