Handling an Elasticsearch Index in the Red State GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state 1 2 ops-pod-loggie-2026.06.11 0 p UNASSIGNED NODE_LEFT ops-pod-loggie-2026.06.11 0 r UNASSIGNED ALLOCATION_FAILED Try to reallocate. 1 POST _cluster/reroute?retry_failed=true Or delete it outright. 1 DELETE ops-pod-loggie-2026.06.11 Scheduled Index Cleanup Keep all indexes for only 7 days. Create an ILM policy 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 PUT _ilm/policy/ops-delete-after-7d { "policy": { "phases": { "hot": { "actions": {} }, "delete": { "min_age": "7d", "actions": { "delete": {} } } } } } Create an index template 1 2 3 4 5 6 7 8 9 10 PUT _index_template/ops-delete-after-7d { "index_patterns": ["*", "-.

1. Symptoms Elasticsearch on Kubernetes was restarting frequently, leaving the service almost unusable. During data import, the Pod’s memory usage kept growing Once the Pod’s memory usage approached the Limit, continued importing triggered an abnormal Pod exit, with the error log ERROR: Elasticsearch exited unexpectedly The Pod’s memory usage did not drop; it stayed near the Limit, and before long the Pod exited abnormally again The Elasticsearch Pod’s memory limit was 64GB, while the JVM memory limit was 32GB.

1. Mount a Memory Storage Directory on the Host Create a directory to mount 1 mkdir /mnt/memory_storage Mount the tmpfs filesystem 1 mount -t tmpfs -o size=800G tmpfs /mnt/memory_storage Storage is allocated on demand, so using 100G of storage only consumes 100G of memory. The host node has 2T of memory, and here 800G is allocated to store Elasticsearch data.

1. Storage Options There are three storage approaches: Isolate by directory and share a single JuiceFS Elasticsearch nodes share one JuiceFS instance, mounting different Elasticsearch nodes through subdirectories. /0/ corresponds to node Node-0 /1/ corresponds to node Node-1 /2/ corresponds to node Node-2 The main advantage of this approach is that it is easy to scale and simple to configure.

If you use Logstash to centrally receive the log input from Filebeat, it easily creates a single point of bottleneck; if you use Kafka to receive the log input from Filebeat, the timeliness of the logs is not guaranteed. Here we output the logs collected by Filebeat directly to Elasticsearch.