Ceph Architecture and Operations

Ceph is a unified distributed storage system that supports block storage (RBD), object storage (RGW), and file storage (CephFS). The official cephadm can orchestrate each daemon as a container (Docker, or Podman + Containerd); see Deploying Ceph in Containers for details.

1. Architecture

1.1 Topology and Data Paths

The diagram below shows a common five-node topology (3 MON + 2 MGR + 5 OSD): MONs only need to come in odd numbers, and in a five-node deployment 3 is enough (tolerating one MON failure); 5 MONs tolerate 2 failures, but the metadata overhead is larger, so there is generally no need to run a MON on every machine. In a single-node validation setup, MON, MGR, and OSD all run on the local machine.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
                         +------------------------+
                         | bootstrap node         |
                         | cephadm / ceph CLI     |
                         +-----------+------------+
                                     |
                          SSH 22 manage nodes / start containers
                                     |
   +--------+   +--------+   +--------+   +--------+   +--------+
   | node1  |   | node2  |   | node3  |   | node4  |   | node5  |
   | MON    |<->| MON    |<->| MON    |   |        |   |        |
   | MGR    |   | MGR    |   |        |   |        |   |        |
   | active |   |standby |   |        |   |        |   |        |
   |RGW(opt)|   |        |   |        |   |        |   |        |
   | OSD    |<->| OSD    |<->| OSD    |<->| OSD    |<->| OSD    |
   +--------+   +--------+   +--------+   +--------+   +--------+
        ^            ^            ^            ^            ^
        +-------- OSD replication / recovery: 6800-7300 ------+

RBD / CSI data path:
  librbd / K8s CSI  -->  MON 3300/6789  -->  CRUSH / PG  -->  OSD 6800-7300

RGW / S3 data path:
  S3 client         -->  RGW 7480       -->  CRUSH / PG  -->  OSD 6800-7300

Management and monitoring path:
  MGR active        -->  Dashboard 8080
  MGR active        -->  Prometheus metrics 9283

Data paths:

RBD / CSI: client → MON (fetch the cluster map) → CRUSH (compute placement) → PG (logical sharding) → OSD (actual reads and writes)
RGW / S3: S3 client → RGW (object gateway) → CRUSH → PG → OSD

CRUSH is Ceph’s data placement algorithm, and a PG is the logical shard between a storage pool and the OSDs.

1.2 Component Purposes and Deployment Requirements

Component	Purpose	Connection / Port	Deployment requirements
cephadm	Orchestrate the cluster, manage nodes	SSH 22	Installed on the bootstrap node; each node needs Docker or Podman
MON	Cluster map, quorum, client entry point	3300 / 6789	Odd number; production usually 3 (5 nodes is plenty too), spread across different nodes
MGR	Management API, Dashboard, metrics modules	8080 / 9283	Usually 2 (active + standby)
OSD	Store data, replicate, recover	6800-7300	Only needed on nodes with block devices; 3 replicas need at least 3 OSDs
RBD	Block storage images	Clients connect to MON/OSD	No separate daemon; just create a pool and an image
RGW	S3 / Swift object gateway	7480	Optional; multiple instances on demand
MDS	CephFS metadata service	6800-7300	Required when deploying CephFS

1.3 Network Ports

The following ports must be reachable between cluster nodes:

Port	Component	Purpose
3300	mon	Monitor v2
6789	mon	Monitor v1 (compatibility)
6800–7300	osd / mds	OSD heartbeat, replication, and recovery
9283	mgr	Prometheus `/metrics`
8080	dashboard	Ceph Dashboard (HTTP)
7480	rgw	S3 / Swift object gateway
9100	node-exp	Host metrics (monitoring stack)

1.4 Version Timeline

Version	Codename	First released	Current status	Recommended OS
v17	Quincy	April 2022	Maintenance (approaching EOL)	RHEL/EL 8, 9; Ubuntu 22.04
v18	Reef	August 2023	Long-term support (LTS)	RHEL/EL 8, 9; Ubuntu 22.04
v19	Squid	July 2024	Current flagship stable release	RHEL/EL 9; Ubuntu 24.04
v20	Tentacle	2025	In development / preview stage	RHEL/EL 9; Ubuntu 24.04
v21	U-codename	Planned for 2026	In planning	Awaiting official release

For compatibility between versions and operating systems, see https://docs.ceph.com/en/latest/start/os-recommendations/

2. Environment Variables

2.1 Multi-Node

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
export CEPH_NODE1_HOST=ceph-node-01
export CEPH_NODE2_HOST=ceph-node-02
export CEPH_NODE3_HOST=ceph-node-03
export CEPH_NODE1_IP=10.0.0.11
export CEPH_NODE2_IP=10.0.0.12
export CEPH_NODE3_IP=10.0.0.13
export CEPH_PUBLIC_NETWORK=10.0.0.0/8
export CEPH_CLUSTER_NETWORK=10.0.0.0/8
export CEPH_DASHBOARD_PORT=8080
export CEPH_IMAGE=quay.io/ceph/ceph:v20.2.2

export RGW_SERVICE=rgw
export RGW_HTTP_PORT=7480

2.2 Single Node

1
2
3
4
5
6
7
8
9
export CEPH_NODE1_HOST=$(hostname)
export CEPH_NODE1_IP=10.0.0.11                    # local external IP, same as bootstrap --mon-ip
export CEPH_PUBLIC_NETWORK=10.0.0.0/8
export CEPH_CLUSTER_NETWORK=10.0.0.0/8
export CEPH_DASHBOARD_PORT=8080
export CEPH_IMAGE=quay.io/ceph/ceph:v20.2.2

export RGW_SERVICE=rgw
export RGW_HTTP_PORT=7480

2.3 Storage Functionality Tests

1
2
3
4
5
6
7
8
export RBD_POOL=test_rbd
export RBD_IMAGE=test_image
export RBD_SIZE=10G

export RGW_UID=testuser
export RGW_DISPLAY_NAME="Test User"

export CEPHFS_NAME=test_cephfs

3. Using Storage

3.1 Block Storage (RBD)

Create a pool and an image

1
2
3
4
ceph osd pool create $RBD_POOL 32 32
ceph osd pool application enable $RBD_POOL rbd
rbd create --size $RBD_SIZE $RBD_POOL/$RBD_IMAGE
rbd ls -p $RBD_POOL

Client mapping, writing, reading

In a multi-node setup, install ceph-common on the client machine and copy the configuration from the management node; in a single-node setup, skip the scp and run the following commands directly on the local machine.

1
2
scp root@$CEPH_NODE1_IP:/etc/ceph/ceph.conf /etc/ceph/
scp root@$CEPH_NODE1_IP:/etc/ceph/ceph.client.admin.keyring /etc/ceph/

Create the RBD image

1
2
3
modprobe rbd
rbd map $RBD_POOL/$RBD_IMAGE
lsblk | grep rbd

Format the block device

1
2
3
mkfs.xfs /dev/rbd0
mkdir -p /mnt/ceph-rbd
mount /dev/rbd0 /mnt/ceph-rbd

Test the file

1
2
echo "rbd test ok" > /mnt/ceph-rbd/test.txt
cat /mnt/ceph-rbd/test.txt

Unmount

1
2
umount /mnt/ceph-rbd
rbd unmap $RBD_POOL/$RBD_IMAGE

3.2 Object Storage (RGW)

Create an S3 user

1
2
3
4
5
6
export RGW_UID=mytest-rgw
export RGW_DISPLAY_NAME=Mytest Rgw

cephadm shell -- radosgw-admin user create \
  --uid=$RGW_UID \
  --display-name="$RGW_DISPLAY_NAME"

Note the access_key and secret_key in the output.

Upload verification (awscli required)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
export CEPH_NODE1_IP=127.0.0.1
export RGW_HTTP_PORT=7480
export AWS_ACCESS_KEY_ID=<access_key>
export AWS_SECRET_ACCESS_KEY=<secret_key>
export RGW_ENDPOINT=http://$CEPH_NODE1_IP:$RGW_HTTP_PORT

echo "rgw test ok" > /tmp/rgw-test.txt
aws --endpoint-url $RGW_ENDPOINT s3 mb s3://test-bucket --region default
aws --endpoint-url $RGW_ENDPOINT s3 cp /tmp/rgw-test.txt s3://test-bucket/
aws --endpoint-url $RGW_ENDPOINT s3 ls s3://test-bucket/

3.3 File Storage (CephFS)

Create a filesystem and deploy the MDS

The MDS stores file layout information.

1
2
3
4
5
6
7
export CEPHFS_NAME=mytest-filestore

ceph fs volume create $CEPHFS_NAME
ceph orch apply mds $CEPHFS_NAME --placement="1 $CEPH_NODE1_HOST"

ceph fs status
ceph orch ps | grep mds

For a single node, replace $CEPH_NODE1_HOST with $(hostname).

Mount, write, read

1
2
3
4
5
mkdir -p /mnt/ceph-cephfs
ceph-fuse /mnt/ceph-cephfs

echo "cephfs test ok" > /mnt/ceph-cephfs/test.txt
cat /mnt/ceph-cephfs/test.txt

Unmount

1
fusermount -u /mnt/ceph-cephfs

4. Routine Inspection

1
2
3
4
5
6
ceph -s
ceph health detail
ceph osd stat
ceph osd tree
ceph df
ceph orch ps

What to watch:

Check item	Normal state	Where to look when abnormal
`health`	`HEALTH_OK`	`ceph health detail` to see the WARN/ERR cause
OSD	All `up` and `in`	Check the disk, container, `ceph orch daemon restart osd.<id>`
PG	`active+clean`	Wait for recovery; if abnormal for a long time, check `ceph pg stat`
MGR	1 active, the rest standby	`ceph mgr dump` to confirm the active/standby switch
Capacity	Utilization < 85%	Add OSDs or clean up data

View recent alerts and logs:

1
2
ceph log last 20
cephadm logs --name mon.$CEPH_NODE1_HOST   # replace with the actual daemon name

5. Configuration Management

Set public network access

1
ceph config set global public_network $CEPH_PUBLIC_NETWORK

Set the cluster network, used by OSD replication, recovery, and backfill; in small environments it can be the same as the public network

1
ceph config set global cluster_network $CEPH_CLUSTER_NETWORK

Reload the MON configuration

1
ceph orch daemon reconfig mon.$CEPH_NODE1_HOST

Disable Dashboard HTTPS

1
ceph config set mgr mgr/dashboard/ssl false

Set the Dashboard HTTP port

1
ceph config set mgr mgr/dashboard/server_port $CEPH_DASHBOARD_PORT

Restart the Dashboard module to apply the configuration

1
2
ceph mgr module disable dashboard
ceph mgr module enable dashboard

View the Dashboard URL and the admin user

1
2
ceph dashboard get-url
ceph dashboard ac-user-show admin

View and roll back configuration:

1
2
3
ceph config dump
ceph config get mon public_network
ceph config rm global some_key    # remove a mistakenly set key

6. Pool and OSD Management

6.1 Storage Pools

1
2
3
4
5
ceph osd pool ls detail
ceph osd pool create <pool> 32 32
ceph osd pool application enable <pool> rbd    # or rgw / cephfs
ceph osd pool set <pool> size 3                # replica count
ceph osd pool set <pool> min_size 2            # minimum writable replicas

Delete a pool (requires the protection flag; the operation is irreversible):

1
ceph osd pool rm <pool> <pool> --yes-i-really-really-mean-it

6.2 Adding OSDs

After preparing raw disks on the new node:

1
2
3
4
ceph orch host add <hostname> <ip>
ceph orch daemon add osd <hostname>:/dev/nvme0n1
# or auto-discover
ceph orch apply osd --all-available-devices

6.3 Decommissioning an OSD

1
2
3
4
5
ceph osd out <osd-id>           # mark out; data migrates away
watch -n 5 ceph -s              # wait for PG recovery to finish
ceph osd crush remove osd.<id>
ceph auth del osd.<id>
ceph osd rm <osd-id>

6.4 OSD Maintenance Mode

Before replacing a disk or performing host maintenance:

1
2
3
ceph osd ok-out <osd-id>        # graceful drain
# after maintenance is complete
ceph osd in <osd-id>

7. Daemons and Node Management

1
2
3
4
5
6
ceph orch host ls
ceph orch ps
ceph orch device ls             # available block devices
ceph orch daemon restart osd.0
ceph orch daemon reconfig mon.*
ceph orch daemon status <daemon-name>

Remove a node from the cluster (first drain the OSDs on it):

1
2
ceph orch host drain <hostname>
ceph orch host rm <hostname>

View orchestration events (when a deployment fails):

1
2
ceph orch ls
ceph -W cephadm                 # watch cephadm events in real time

8. RGW Object Gateway

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# multi-node: 2 instances for high availability
ceph orch apply rgw $RGW_SERVICE \
  --placement="2 $CEPH_NODE1_HOST $CEPH_NODE2_HOST" \
  --port=$RGW_HTTP_PORT

# single node
ceph orch apply rgw $RGW_SERVICE \
  --placement="1 $(hostname)" \
  --port=$RGW_HTTP_PORT

ceph orch ps

User and bucket management:

1
2
3
radosgw-admin user list
radosgw-admin user info --uid=<uid>
radosgw-admin user rm --uid=<uid>

9. Metrics Monitoring

Ceph MGR has a built-in Prometheus exporter (:9283); just add a scrape job in Prometheus.

9.1 Enable the MGR Prometheus Module

1
2
ceph mgr module enable prometheus
ceph mgr services | grep prometheus

9.2 Prometheus Job Configuration

Append the following under scrape_configs in prometheus.yml (the IPs correspond to $CEPH_NODE1_IP and so on from §2.1):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
- job_name: ceph-mgr
  metrics_path: /metrics
  static_configs:
    - targets:
        - 10.0.0.11:9283    # $CEPH_NODE1_IP
        - 10.0.0.12:9283    # $CEPH_NODE2_IP
      labels:
        cluster: ceph_cluster_3n
        component: mgr

- job_name: node-exporter
  static_configs:
    - targets:
        - 10.0.0.11:9100    # $CEPH_NODE1_IP
        - 10.0.0.12:9100    # $CEPH_NODE2_IP
        - 10.0.0.13:9100    # $CEPH_NODE3_IP
      labels:
        cluster: ceph_cluster_3n

For a single node, keep only $CEPH_NODE1_IP in targets. After modifying, reload the Prometheus configuration:

1
2
curl -X POST http://<prometheus-ip>:9090/-/reload    # requires --web.enable-lifecycle
# or restart the Prometheus process / container

Verify:

1
curl -s http://$CEPH_NODE1_IP:9283/metrics | grep ceph_cluster_total_bytes

In the Prometheus UI under Status → Targets, confirm that ceph-mgr and node-exporter are UP.

9.3 Metrics to Watch

Metric	Meaning
`ceph_cluster_total_bytes`	Total cluster capacity
`ceph_cluster_total_used_bytes`	Used capacity
`ceph_osd_up`	Whether the OSD is online (1 = normal)
`ceph_osd_in`	Whether the OSD is in the CRUSH tree
`ceph_pg_active`	Number of active PGs
`ceph_health_status`	Cluster health (0 = OK)

9.4 Importing Grafana Dashboards

Grafana Dashboards → Import; recommended dashboards:

Dashboard	ID	Description
Ceph Cluster	2842	Cluster capacity, OSD, PG, performance
Ceph - RadosGW	5336	RGW requests and bandwidth (after deploying RGW)

10. Integrating with Grafana / Dashboard

10.1 Ceph Dashboard

1
2
ceph dashboard get-url
ceph dashboard ac-user-show admin

10.2 External Grafana Data Sources

Data source	URL
MGR Prometheus	`http://<active-mgr-ip>:9283/`
cephadm Prometheus	`http://<prometheus-ip>:9095/` (if the monitoring stack was not skipped)
Prometheus	`http://<prometheus-ip>:9090/`

11. Upgrading and Uninstalling

11.1 Switching Image Versions

1
2
3
ceph orch upgrade ls              # list available versions
ceph orch upgrade start --image $CEPH_IMAGE
ceph orch upgrade status

11.2 Uninstalling the Cluster

Run this only in a test environment; it deletes all data:

1
2
3
4
5
ceph orch rm --force rgw
ceph orch rm --force mds
ceph osd purge <id> --yes-i-really-mean-it
# once all OSDs are cleaned up
cephadm rm-cluster --fsid $(ceph fsid) --force

12. Common Issues

Docker’s Seccomp restriction causes a pthread_create error

1
2
3
docker run --rm --ipc=host --net=host \
  --entrypoint /usr/bin/ceph-authtool \
  $CEPH_IMAGE --gen-print-key

It fails with

1
2
/usr/bin/ceph-authtool: stderr Thread::try_create(): pthread_create failed with error 1/ceph/rpmbuild/BUILD/ceph-20.2.2/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f2c842500c0 time 2026-07-12T01:12:23.852165+0000
/usr/bin/ceph-authtool: stderr /ceph/rpmbuild/BUILD/ceph-20.2.2/src/common/Thread.cc: 165: FAILED ceph_assert(ret == 0)

Disable Seccomp

1
2
3
4
docker run --rm --ipc=host --net=host \
  --security-opt seccomp=unconfined \
  --entrypoint /usr/bin/ceph-authtool \
  $CEPH_IMAGE --gen-print-key

Permanent fix: edit /usr/sbin/cephadm and, after

1
2
if self.host_network:
   cmd_args.append('--net=host')

append

1
cmd_args.extend(['--security-opt', 'seccomp=unconfined'])

View the error log

1
tail -200 /var/log/ceph/cephadm.log | grep -i "error\|fail"

Creating a service fails with EPERM Operation

1
Thread::try_create(): pthread_create failed with error 1 → EPERM Operation not permitted

Tentacle RGW creates a SCHED_FIFO real-time scheduling thread by default, and Docker’s default seccomp blacklist blocks that system call

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
service_type: rgw
service_id: default
placement:
  hosts:
    - my-hostname
spec:
  rgw_frontend_port: 7480
  rgw_frontend_type: beast
extra_container_args:
  - --security-opt
  - seccomp=unconfined
  - --ulimit
  - nproc=65535:65535
  - --ulimit
  - nofile=1048576:1048576

1
ceph orch apply -i rgw-spec.yaml