Deploying Ceph in Containers

This article covers single-node validation and three-node production deployment with cephadm. For architecture, operations, and functional testing, see Ceph Architecture and Operations.

1. Disk Preparation

An OSD can only use a block device (/dev/sdX, a partition, /dev/loopN).

1.1 Inspecting Available Devices

1
2
3
4
5
6
7
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL

# 文件系统与 UUID
lsblk -f

# 指定盘是否已挂载（无输出 = 未挂载）
findmnt /dev/nvme0n1

1.2 Dedicated Data Disks

Taking /dev/nvme0n1 as an example, run the following on every node that has a data disk:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
export OSD_DISK=/dev/nvme0n1

# 再次确认：容量、挂载、分区
lsblk $OSD_DISK
mount | grep -q "$OSD_DISK" && echo "磁盘已挂载，请先 umount" && exit 1

# 清除旧分区表与文件系统签名（破坏性操作，务必确认盘符）
wipefs -a $OSD_DISK

# 验证：FSTYPE 应为空
lsblk -f $OSD_DISK
blkid $OSD_DISK || true    # 无输出表示无签名

If you need a whole disk as a single partition before handing it to the OSD:

1
2
3
4
parted -s $OSD_DISK mklabel gpt mkpart primary 0% 100%
partprobe $OSD_DISK
export OSD_PART=${OSD_DISK}1    # 如 /dev/nvme0n11
lsblk -f $OSD_PART

2. Single-Node Deployment

2.1 Configuring Environment Variables

1
2
3
4
5
6
7
8
9
export CEPH_RELEASE=20.2.2
export CEPH_CODENAME=tentacle                           # 20.x → tentacle；19.x → squid；18.x → reef；17.x → quincy
export CEPH_IMAGE=quay.io/ceph/ceph:v${CEPH_RELEASE}
export CEPH_CLUSTER_NAME=ceph_single
export CEPH_NODE_IP=10.0.0.11                      # 本机对外可被访问的 IP
export CEPH_PUBLIC_NETWORK=10.0.0.0/8           # bootstrap 后写入，覆盖整个 10 内网
export CEPH_CLUSTER_NETWORK=10.0.0.0/8
export CEPH_DASHBOARD_PORT=8080
export CEPH_DATA=/data/ops/ceph/$CEPH_CLUSTER_NAME

2.2 Installing cephadm

1
2
3
4
5
6
7
8
9
mkdir -p $CEPH_DATA

curl --silent --remote-name --location \
  https://download.ceph.com/rpm-${CEPH_RELEASE}/el9/noarch/cephadm
chmod +x cephadm

./cephadm add-repo --version $CEPH_RELEASE
./cephadm install
./cephadm install ceph-common

1
cephadm --image $CEPH_IMAGE version

2.3 Bootstrapping the Cluster

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
cat <<EOF > $CEPH_DATA/single-node.conf
[global]
# 默认副本数为 1
osd_pool_default_size = 1
# 最小副本数为 1
osd_pool_default_min_size = 1
# 允许创建大小为 1 的池
mon_allow_pool_size_one = true
# 对于单节点，禁用一些不必要的警告
mgr_allow_pool_size_one = true
EOF

1
2
3
4
5
cephadm --image $CEPH_IMAGE bootstrap \
  --mon-ip $CEPH_NODE_IP \
  --config $CEPH_DATA/single-node.conf \
  --skip-mon-network \
  --allow-fqdn-hostname

1
ceph config set global public_network ${CEPH_PUBLIC_NETWORK}

2.4 Deploying OSDs

Inspect disks

1
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT

Add a disk as an OSD

1
ceph orch daemon add osd $(hostname):/dev/vdd

2.5 Enabling the RGW Gateway

Set environment variables

1
2
export RGW_SERVICE=default
export RGW_HTTP_PORT=7480

Enable the gateway

1
2
3
ceph orch apply rgw $RGW_SERVICE \
  --placement="1 $(hostname)" \
  --port=$RGW_HTTP_PORT

Inspect the service

1
ceph orch ps --daemon-type rgw

2.6 Verifying the Cluster

Check service status

1
ceph orch ls

1
2
3
4
5
6
7
8
9
NAME          PORTS        RUNNING  REFRESHED  AGE  PLACEMENT
alertmanager  ?:9093,9094      1/1  3m ago     3d   count:1
crash                          1/1  3m ago     12h  count:1
grafana       ?:3000           1/1  3m ago     3d   count:1
mgr                            1/1  3m ago     16h  count:1
mon                            1/1  3m ago     16h  count:1
osd.default                      1  3m ago     3d   my-hostname
prometheus    ?:9095           1/1  3m ago     3d   count:1
rgw.default   ?:7480           0/1  3m ago     3m   my-hostname;count:1

Check storage status

1
2
3
ceph -s
ceph df
ceph health detail

3. Three-Node Deployment

For node planning, see Ceph Architecture and Operations. Complete the entire flow — environment variables, passwordless SSH, installing cephadm, bootstrap, and host adoption — independently on all three machines.

3.1 Configuring Environment Variables

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
export CEPH_RELEASE=20.2.2
export CEPH_CODENAME=tentacle                           # 20.x → tentacle；19.x → squid；18.x → reef；17.x → quincy
export CEPH_IMAGE=quay.io/ceph/ceph:v${CEPH_RELEASE}
export CEPH_CLUSTER_NAME=ceph_cluster_3n

export CEPH_NODE1_HOST=ceph-node-01
export CEPH_NODE2_HOST=ceph-node-02
export CEPH_NODE3_HOST=ceph-node-03

export CEPH_NODE1_IP=10.0.0.11
export CEPH_NODE2_IP=10.0.0.12
export CEPH_NODE3_IP=10.0.0.13

export CEPH_NODE_IP=$CEPH_NODE1_IP
export CEPH_PUBLIC_NETWORK=10.0.0.0/8           # bootstrap 后写入，覆盖整个 10 内网
export CEPH_CLUSTER_NETWORK=10.0.0.0/8
export CEPH_DASHBOARD_PORT=8080
export CEPH_DATA=/data/ops/ceph/$CEPH_CLUSTER_NAME

3.2 Setting Up Passwordless SSH

1
2
3
4
5
6
7
[ -f ~/.ssh/id_rsa ] || ssh-keygen -t rsa -b 4096 -N "" -f ~/.ssh/id_rsa

ssh-copy-id -o StrictHostKeyChecking=no root@$CEPH_NODE2_IP
ssh-copy-id -o StrictHostKeyChecking=no root@$CEPH_NODE3_IP

ssh -o BatchMode=yes root@$CEPH_NODE2_IP hostname
ssh -o BatchMode=yes root@$CEPH_NODE3_IP hostname

3.3 Installing cephadm on All Nodes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
mkdir -p $CEPH_DATA

curl --silent --remote-name --location \
  https://download.ceph.com/rpm-${CEPH_RELEASE}/el9/noarch/cephadm
chmod +x cephadm

./cephadm rm-repo 2>/dev/null || true
./cephadm add-repo --version $CEPH_RELEASE
./cephadm install
./cephadm install ceph-common

which cephadm

The following operations only need to be run on the admin node.

3.4 Bootstrapping the Cluster

cephadm bootstrap validates that public_network must be the subnet of a local NIC, so you cannot write 10.0.0.0/8 directly during bootstrap. First complete bootstrap with --skip-mon-network, then write the entire 10.x private network segment:

1
2
3
4
cephadm --image $CEPH_IMAGE bootstrap \
  --mon-ip $CEPH_NODE_IP \
  --skip-mon-network \
  --allow-fqdn-hostname

3.5 Adopting Other Nodes

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
ssh-keyscan -H $CEPH_NODE2_IP $CEPH_NODE3_IP >> ~/.ssh/known_hosts

ceph orch host add $CEPH_NODE2_HOST $CEPH_NODE2_IP
ceph orch host add $CEPH_NODE3_HOST $CEPH_NODE3_IP

ceph orch host label add $CEPH_NODE1_HOST _admin
ceph orch host label add $CEPH_NODE2_HOST _admin
ceph orch host label add $CEPH_NODE3_HOST _admin

ceph orch host ls

3.6 Deploying MON / MGR / OSD

Run on node1 (the admin node):

1
2
ceph orch apply mon --placement="3 $CEPH_NODE1_HOST $CEPH_NODE2_HOST $CEPH_NODE3_HOST"
ceph orch apply mgr --placement="2 $CEPH_NODE1_HOST $CEPH_NODE2_HOST"

Once each node has a bare disk ready, run on the admin node:

1
ceph orch apply osd --all-available-devices

Or specify explicitly:

1
2
3
ceph orch daemon add osd $CEPH_NODE1_HOST:/dev/nvme0n1
ceph orch daemon add osd $CEPH_NODE2_HOST:/dev/nvme0n1
ceph orch daemon add osd $CEPH_NODE3_HOST:/dev/nvme0n1

Check orchestration progress:

1
2
ceph orch ps
watch -n 5 ceph -s

3.7 Verifying the Cluster

1
2
3
4
5
6
ceph -s
ceph mon stat
ceph osd tree
ceph df
ceph health detail
ceph orch host ls

Expected:

mon: 3 daemons
mgr: ceph-node-01(active), ceph-node-02(standby) or similar
The osd daemons on each node are all up
health: HEALTH_OK

3.8 Printing the Delivery Result

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
FSID=$(ceph fsid)
cat <<EOF
Ceph Cluster:
  Cluster Name:     $CEPH_CLUSTER_NAME
  FSID:             $FSID
  Nodes:
    $CEPH_NODE1_HOST: $CEPH_NODE1_IP
    $CEPH_NODE2_HOST: $CEPH_NODE2_IP
    $CEPH_NODE3_HOST: $CEPH_NODE3_IP
  Public Network:   $CEPH_PUBLIC_NETWORK
  Cluster Network:  $CEPH_CLUSTER_NETWORK
  Image:            $CEPH_IMAGE
  Config:           /etc/ceph/ceph.conf
  Keyring:          /etc/ceph/ceph.client.admin.keyring
  Dashboard:        $(ceph dashboard get-url 2>/dev/null || echo "ceph dashboard get-url")
  MGR Prometheus:   http://$CEPH_NODE1_IP:9283/metrics
EOF