Operations
Ceph Architecture and Operations
· β˜• 11 min read
Ceph is a unified distributed storage system that supports block storage (RBD), object storage (RGW), and file storage (CephFS). The official cephadm can orchestrate each daemon as a container (Docker, or Podman + Containerd); see Deploying Ceph in Containers for details. 1. Architecture 1.1 Topology and Data Paths The diagram below shows a common five-node topology (3 MON + 2 MGR + 5 OSD): MONs only need to come in odd numbers, and in a five-node deployment 3 is enough (tolerating one MON failure); 5 MONs tolerate 2 failures, but the metadata overhead is larger, so there is generally no need to run a MON on every machine.

Deploying VictoriaMetrics in Containers
· β˜• 8 min read
VictoriaMetrics (VM for short) is a time-series database compatible with the Prometheus ecosystem. It uses fewer resources than Prometheus and comes in both single-node and cluster editions. The single-node edition can directly replace Prometheus storage; the cluster edition splits reads, writes, and storage across vmstorage / vminsert / vmselect, giving it stronger horizontal scaling.

How to Separate Events from Kubernetes to a Dedicated Etcd
· β˜• 2 min read
The disk pressure on the original etcd is reduced to below 5%, the risk factor is very low, and the cluster shows no anomalies. 1. Deploying the Events Etcd Cluster Events have lower consistency requirements than the main Etcd. Deploying a 3-node cluster is recommended. SSD is recommended for the disk, and the data directory should be kept separate from the main Etcd.

Inference Application Troubleshooting Checklist
· β˜• 1 min read
Pod View the child processes of PID 1 1 ps --ppid 1 -f View TIME_WAIT connections 1 netstat -anp | grep TIME_WAIT Istio Ingress 1 kubectl -n istio-ingress logs -l app=istio-ingress |grep vlm-tag-v1 EastWestGateway 1 kubectl -n istio-ingress logs -l app=istio-eastwestgateway |grep vlm-tag-v1 Slow XDS Push 1 2 3 4 5 histogram_quantile(0.

Elasticsearch troubleshooting
· β˜• 2 min read
Handling an Elasticsearch Index in the Red State GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state 1 2 ops-pod-loggie-2026.06.11 0 p UNASSIGNED NODE_LEFT ops-pod-loggie-2026.06.11 0 r UNASSIGNED ALLOCATION_FAILED Try to reallocate. 1 POST _cluster/reroute?retry_failed=true Or delete it outright. 1 DELETE ops-pod-loggie-2026.06.11 Scheduled Index Cleanup Keep all indexes for only 7 days. Create an ILM policy 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 PUT _ilm/policy/ops-delete-after-7d { "policy": { "phases": { "hot": { "actions": {} }, "delete": { "min_age": "7d", "actions": { "delete": {} } } } } } Create an index template 1 2 3 4 5 6 7 8 9 10 PUT _index_template/ops-delete-after-7d { "index_patterns": ["*", "-.