Ceph in Kubernetes

1. Different Kinds of Storage

1.1 File Storage

File storage is storage based on files. When accessing data, you need to provide the corresponding lookup path.

It is suitable for services such as FTP and NFS.

1.2 Block Storage

Block storage splits data into blocks and stores each part separately. When accessing data, the underlying storage software reassembles these scattered pieces.

Block storage is usually deployed in a storage area network (SAN), which makes for fast retrieval and easy use and management.

It is suitable for services such as disk arrays and hard drives.

1.3 Object Storage

Object storage breaks data down into discrete units called objects, kept in a single repository. Each object storage volume is a self-contained database, and users can look up objects and metadata in a distributed manner.

Object storage requires the object to be written in full in one go, and the object cannot be modified.

It is suitable for large-capacity distributed storage such as Ceph.

2. Ceph

Ceph is a highly reliable, scalable distributed storage system designed around commodity hardware, and it supports all three kinds of storage at once: file storage, block storage, and object storage.

Ceph is built on RADOS at its lower layer, and RADOS is itself a complete distributed object storage system.

2.1 Ceph’s Core Components

MON monitoring, which monitors cluster status and maintains charts that display cluster status, such as the OSD Map, Monitor Map, PG Map, and CRUSH Map.
MDS (optional) metadata service, which stores the file system’s metadata and manages the directory structure.
OSD storage service, which stores, replicates, balances, and recovers data, and performs heartbeat checks with other OSDs. Usually one disk corresponds to one OSD.

2.2 Ceph’s Storage Process

Ceph uses the CRUSH algorithm for storage at its lower layer. As shown in the figure above, the storage process is as follows:

First, the data is split into multiple Objects. An Object is the smallest storage unit in Ceph. Each Object has an Object id, and the size is configurable, defaulting to 4 MB.
Second, to reduce the index table from Objects to OSDs, Placement Groups (PGs) are introduced for management. An Object is mapped to a PG through hashing, and each PG contains multiple Objects.
Third, the PG is mapped to an OSD through the CRUSH algorithm. With multiple replicas, each PG is mapped to multiple OSDs.

2.3 Deploying Ceph

ceph-deploy is the deployment tool provided by the Ceph project. It logs in to other nodes remotely over SSH and then runs commands to complete the deployment.

Typically, deploying a Ceph cluster takes the following steps:

Plan the cluster nodes
Configure users, passwordless SSH, and hosts
Install ceph-deploy and ceph
Deploy monitors, managers, MDS, and OSDs

3. Ceph in Kubernetes

3.1 The Problem with Using PV/PVC Directly

For an introduction to PV and PVC, see PV, PVC.

The administrator has to pre-allocate PVs. When a user creates a PVC to request storage, Kubernetes tries to match the PVC against the pre-allocated PVs. If a match is found, the PVC is bound to the PV and made available to the user.

Under this matching strategy, the administrator cannot precisely meet each user’s needs, and can only keep resizing the PVs. This places a heavy burden on storage operations.

To address this, Kubernetes 1.6 introduced dynamic provisioning, StorageClass, and Provisioner. The administrator uses a StorageClass to describe the storage being offered. A StorageClass contains a set of parameter information such as capacity and IOPS. The storage vendor’s Provisioner dynamically allocates PVs according to the parameters set in the StorageClass.

3.2 Provisioner

When a user requests storage resources through a PVC, the StorageClass uses the Provisioner to automatically create the PV the user needs.

The detailed flow in the figure above is analyzed as follows:

(1) The Pod mounts the storage volume and requests a PVC

(2) The PVC finds the storage class StorageClass according to the storage type (rbd in this case)

(3) The Provisioner dynamically generates a persistent volume PV according to the StorageClass

(4) The persistent volume PV and the PVC finally form a binding relationship

(5) The persistent volume PV starts to be provided for the Pod’s use

By configuring different Provisioners, a StorageClass supports many types of storage volumes. Quite a few kinds are supported so far: AWSElasticBlockStore, AzureFile, AzureDisk, Cinder, Flocker, GCEPersistentDisk, Glusterfs, PhotonPersistentDisk, Quobyte, RBD, VsphereVolume, PortworxVolume, ScaleIO, StorageOS, and others.

Besides the types Kubernetes supports natively, more storage types can be supported by installing extension plugins. For example, nfs-client-provisioner for NFS.

Ceph provides a Provisioner to the StorageClass for creating PVs through RBD.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/rbd
parameters:
  monitors: 10.16.153.105:6789
  adminId: kube
  adminSecretName: ceph-secret
  adminSecretNamespace: kube-system
  pool: kube
  userId: kube
  userSecretName: ceph-secret-user
  userSecretNamespace: default
  fsType: ext4
  imageFormat: "2"
  imageFeatures: "layering"