1. Local Storage Capacity
Required disk size (GB) = data retention period _ samples fetched per second _ metric data size / 1024 / 1024 / 1024
Where
- Samples fetched per second: rate(prometheus_tsdb_head_samples_appended_total[1d])
- Average size of a sample within one hour: rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d])/rate(prometheus_tsdb_compaction_chunk_samples_sum[1d])
The disk consumption for one day (86400 seconds) can be queried directly in Prometheus:
86400 * (rate(prometheus_tsdb_head_samples_appended_total[1d]) * (rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[1d]))) / 1024 /1024 / 1024
For example, if it returns {instance="localhost:9090", job="prometheus"} 4.437027408140867, that means the localhost:9090 instance needs to consume 4.437 GB of storage per day. At the same time, in an instance there are no fewer than 3 WAL files used to store raw data, each 128 MB.
2. Memory Consumption
Memory consumption = the memory consumption of the Prometheus Server itself + the memory consumption of data blocks + the memory consumption of scraping metrics + the memory consumption caused by queries
- The memory consumption of the Prometheus Server itself
On a freshly installed multi-node high-availability cluster, the memory consumption of the Prometheus Server is around 500 MB.
- The memory consumption of data blocks
It is mainly related to the following parameters
- Samples fetched per second: rate(prometheus_tsdb_head_samples_appended_total[1d])
- The average number of labels per metric
- The total number of distinct label pairs
- The average size of each label pair
- The flush-to-disk cycle of data blocks
- The memory consumption of scraping metrics
It is mainly related to the following parameters
- Samples fetched per second: rate(prometheus_tsdb_head_samples_appended_total[1d])
- The average size of a sample within one hour
- The scrape interval, usually 15s
The two parts above can be estimated on the page https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion/.
- The memory consumption caused by queries
When the queried data is not in memory, Prometheus loads data from the disk into memory, which brings extra memory consumption.
In production, the average memory consumption across more than 40 clusters, queried with avg(container_memory_working_set_bytes{image!="", container="prometheus-server"}) / 1024 /1024, is 953 MB, with an average of 300 Pods per cluster.
