Deploying VictoriaMetrics in Containers

VictoriaMetrics (VM for short) is a time-series database compatible with the Prometheus ecosystem. It uses fewer resources than Prometheus and comes in both single-node and cluster editions. The single-node edition can directly replace Prometheus storage; the cluster edition splits reads, writes, and storage across vmstorage / vminsert / vmselect, giving it stronger horizontal scaling.

1. VictoriaMetrics Single Node

The single-node edition (victoria-metrics) combines ingestion, storage, and querying in one process. It suits small-to-medium monitoring setups and works as a Prometheus remote_write remote storage target.

1.1 Configure Environment Variables

1
2
3
4
5
6
7
export CONTAINER_CLI=nerdctl
export VM_TAG=v1.147.0
export VM_IMAGE=victoriametrics/victoria-metrics:$VM_TAG
export VM_INSTANCE_NAME=victoriametrics
export VM_DATA=/data/ops/victoriametrics/$VM_INSTANCE_NAME
export VM_HTTP_PORT=8428
export VM_RETENTION=15d

1.2 Start the Service

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
mkdir -p $VM_DATA/data

$CONTAINER_CLI run -d \
  --name $VM_INSTANCE_NAME \
  --restart always \
  --security-opt apparmor=unconfined \
  --security-opt seccomp=unconfined \
  --network host \
  --ulimit nofile=1048576:1048576 \
  --memory-swappiness=0 \
  -v $VM_DATA/data:/victoria-metrics-data \
  $VM_IMAGE \
  -storageDataPath=/victoria-metrics-data \
  -retentionPeriod=$VM_RETENTION \
  -httpListenAddr=:$VM_HTTP_PORT

1.3 Verify

1
2
3
4
5
6
7
8
# Health check
curl -s http://127.0.0.1:$VM_HTTP_PORT/health

# Inspect its own metrics
curl -s http://127.0.0.1:$VM_HTTP_PORT/metrics | head

# PromQL query (compatible with the Prometheus API)
curl -s 'http://127.0.0.1:8428/api/v1/query?query=up'

Write test (Prometheus remote_write format):

1
2
3
curl -s -X POST http://127.0.0.1:$VM_HTTP_PORT/api/v1/import/prometheus -d '
vm_test_metric{env="dev"} 1
'

1.4 Print the Delivery Result

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
cat <<EOF
VictoriaMetrics (single):
  Instance Name:  $VM_INSTANCE_NAME
  IP:             $(hostname -I | awk '{print $1}')
  HTTP Port:      $VM_HTTP_PORT
  Retention:      $VM_RETENTION
  Write URL:      http://$(hostname -I | awk '{print $1}'):$VM_HTTP_PORT/api/v1/write
  Query URL:      http://$(hostname -I | awk '{print $1}'):$VM_HTTP_PORT/api/v1/query
  Metrics URL:    http://$(hostname -I | awk '{print $1}'):$VM_HTTP_PORT/metrics
EOF

2. VictoriaMetrics Cluster

The example below uses 3 vmstorage nodes + vminsert / vmselect running on each. This is the minimal production topology for the cluster edition: vmstorage stores data, vminsert handles write routing, and vmselect handles query aggregation.

Node	IP (example)	Role
node1	10.0.0.1	vmstorage + vminsert + vmselect
node2	10.0.0.2	vmstorage + vminsert + vmselect
node3	10.0.0.3	vmstorage + vminsert + vmselect

All three machines must be able to reach each other on the following ports:

Port	Component	Purpose
8400	vmstorage	Receives writes from vminsert (TCP)
8401	vmstorage	Receives queries from vmselect (TCP)
8480	vminsert	HTTP writes / remote_write
8481	vmselect	HTTP queries / Grafana data source
8482	vmstorage	HTTP monitoring `/metrics`

2.1 Configure Environment Variables on Each Node

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
export CONTAINER_CLI=nerdctl
export VM_TAG=v1.147.0-cluster
export VM_CLUSTER_NAME=vm_cluster_3n

export VM_STORAGE_IMAGE=victoriametrics/vmstorage:$VM_TAG
export VM_INSERT_IMAGE=victoriametrics/vminsert:$VM_TAG
export VM_SELECT_IMAGE=victoriametrics/vmselect:$VM_TAG

export VM_NODE1_IP=10.0.0.1
export VM_NODE2_IP=10.0.0.2
export VM_NODE3_IP=10.0.0.3

export NODE_ID=1                              # node2 uses 2, node3 uses 3
export VM_NODE_IP=$VM_NODE1_IP                # local IP: node1/2/3 fill in the matching variable

export VM_STORAGE_PORT=8400
export VM_SELECT_PORT=8401
export VM_INSERT_HTTP_PORT=8480
export VM_SELECT_HTTP_PORT=8481
export VM_STORAGE_HTTP_PORT=8482
export VM_RETENTION=15d
export VM_REPLICATION_FACTOR=2                # replication factor, no greater than the number of vmstorage nodes

export VM_DATA=/data/ops/victoriametrics/$VM_CLUSTER_NAME-node$NODE_ID
export VM_STORAGE_NODES="$VM_NODE1_IP:$VM_STORAGE_PORT,$VM_NODE2_IP:$VM_STORAGE_PORT,$VM_NODE3_IP:$VM_STORAGE_PORT"
export VM_SELECT_NODES="$VM_NODE1_IP:$VM_SELECT_PORT,$VM_NODE2_IP:$VM_SELECT_PORT,$VM_NODE3_IP:$VM_SELECT_PORT"

Check whether the ports are in use:

1
lsof -i :$VM_STORAGE_PORT -i :$VM_SELECT_PORT -i :$VM_INSERT_HTTP_PORT -i :$VM_SELECT_HTTP_PORT -i :$VM_STORAGE_HTTP_PORT

2.2 Start vmstorage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
mkdir -p $VM_DATA/vmstorage

$CONTAINER_CLI run -d \
  --name vmstorage-$VM_CLUSTER_NAME-$NODE_ID \
  --restart always \
  --security-opt apparmor=unconfined \
  --security-opt seccomp=unconfined \
  --network host \
  --ulimit nofile=1048576:1048576 \
  --memory-swappiness=0 \
  -v $VM_DATA/vmstorage:/storage \
  $VM_STORAGE_IMAGE \
  -storageDataPath=/storage \
  -retentionPeriod=$VM_RETENTION \
  -vminsertAddr=:$VM_STORAGE_PORT \
  -vmselectAddr=:$VM_SELECT_PORT \
  -httpListenAddr=:$VM_STORAGE_HTTP_PORT

2.3 Start vminsert

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
$CONTAINER_CLI run -d \
  --name vminsert-$VM_CLUSTER_NAME-$NODE_ID \
  --restart always \
  --security-opt apparmor=unconfined \
  --security-opt seccomp=unconfined \
  --network host \
  --ulimit nofile=1048576:1048576 \
  --memory-swappiness=0 \
  $VM_INSERT_IMAGE \
  -storageNode=$VM_STORAGE_NODES \
  -replicationFactor=$VM_REPLICATION_FACTOR \
  -httpListenAddr=:$VM_INSERT_HTTP_PORT

2.4 Start vmselect

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$CONTAINER_CLI run -d \
  --name vmselect-$VM_CLUSTER_NAME-$NODE_ID \
  --restart always \
  --security-opt apparmor=unconfined \
  --security-opt seccomp=unconfined \
  --network host \
  --ulimit nofile=1048576:1048576 \
  --memory-swappiness=0 \
  $VM_SELECT_IMAGE \
  -storageNode=$VM_SELECT_NODES \
  -httpListenAddr=:$VM_SELECT_HTTP_PORT

2.5 Verify the Cluster

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# vmstorage metrics
curl -s http://$VM_NODE_IP:$VM_STORAGE_HTTP_PORT/metrics | grep vm_rows | head

# Write through vminsert
curl -s -X POST "http://$VM_NODE_IP:$VM_INSERT_HTTP_PORT/insert/0/prometheus/api/v1/import/prometheus" -d '
vm_cluster_test{node="'"$NODE_ID"'"} 1
'

# Query through vmselect
curl -s "http://$VM_NODE_IP:$VM_SELECT_HTTP_PORT/select/0/prometheus/api/v1/query?query=vm_cluster_test"

Check whether every vmstorage node is seen by vminsert / vmselect:

1
2
curl -s http://$VM_NODE_IP:$VM_INSERT_HTTP_PORT/metrics | grep vm_rpc_vmstorage_is_reachable
curl -s http://$VM_NODE_IP:$VM_SELECT_HTTP_PORT/metrics | grep vm_rpc_vmstorage_is_reachable

A value of 1 for vm_rpc_vmstorage_is_reachable means connectivity is fine.

2.6 Print the Delivery Result

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
cat <<EOF
VictoriaMetrics Cluster:
  Cluster Name:       $VM_CLUSTER_NAME
  Node ID:            $NODE_ID
  Node IPs:
    node1:            $VM_NODE1_IP
    node2:            $VM_NODE2_IP
    node3:            $VM_NODE3_IP
  Retention:          $VM_RETENTION
  Replication Factor: $VM_REPLICATION_FACTOR
  Write URLs (any vminsert):
    http://$VM_NODE1_IP:$VM_INSERT_HTTP_PORT/insert/0/prometheus/api/v1/write
    http://$VM_NODE2_IP:$VM_INSERT_HTTP_PORT/insert/0/prometheus/api/v1/write
    http://$VM_NODE3_IP:$VM_INSERT_HTTP_PORT/insert/0/prometheus/api/v1/write
  Query URLs (any vmselect):
    http://$VM_NODE1_IP:$VM_SELECT_HTTP_PORT/select/0/prometheus/
    http://$VM_NODE2_IP:$VM_SELECT_HTTP_PORT/select/0/prometheus/
    http://$VM_NODE3_IP:$VM_SELECT_HTTP_PORT/select/0/prometheus/
EOF

3. vmagent Scrape Configuration

vmagent scrapes Exporter metrics and remote_writes them to VM. It can replace the scraping side of Prometheus. Point it at vmsingle for the single-node edition; point it at any vminsert for the cluster edition.

3.1 Configure Environment Variables

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
export CONTAINER_CLI=nerdctl
export VM_TAG=v1.147.0
export VM_AGENT_IMAGE=victoriametrics/vmagent:$VM_TAG
export VM_AGENT_NAME=vmagent
export VM_AGENT_DATA=/data/ops/victoriametrics/$VM_AGENT_NAME
export VM_AGENT_HTTP_PORT=8429

# Single-node edition
export VM_REMOTE_WRITE_URL=http://127.0.0.1:8428/api/v1/write

# Cluster edition (switch to a vminsert address)
export VM_REMOTE_WRITE_URL=http://10.0.0.1:8480/insert/0/prometheus/api/v1/write

3.2 Generate the Scrape Configuration

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
mkdir -p $VM_AGENT_DATA

cat > $VM_AGENT_DATA/scrape.yml <<'EOF'
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: node-exporter
    static_configs:
      - targets:
          - 10.0.0.1:9100
EOF

3.3 Start vmagent

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
$CONTAINER_CLI run -d \
  --name $VM_AGENT_NAME \
  --restart always \
  --security-opt apparmor=unconfined \
  --security-opt seccomp=unconfined \
  --network host \
  --ulimit nofile=1048576:1048576 \
  --memory-swappiness=0 \
  -v $VM_AGENT_DATA/scrape.yml:/etc/vmagent/scrape.yml \
  $VM_AGENT_IMAGE \
  -promscrape.config=/etc/vmagent/scrape.yml \
  -remoteWrite.url=$VM_REMOTE_WRITE_URL \
  -httpListenAddr=:$VM_AGENT_HTTP_PORT

3.4 Verify

1
2
curl -s http://127.0.0.1:$VM_AGENT_HTTP_PORT/metrics | grep vmagent_remotewrite_rows_sent_total
curl -s http://127.0.0.1:$VM_AGENT_HTTP_PORT/targets

4. Self-Monitoring

Every VM component exposes /metrics. It is best to have vmagent scrape them all and write them into the cluster.

4.1 Scrape Configuration

Have vmagent scrape the VM components and node-exporter together, and write them into the cluster via remote_write. This replaces the configuration from section 3.2 and covers the components on all three nodes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
export VM_AGENT_DATA=/data/ops/victoriametrics/vmagent
mkdir -p $VM_AGENT_DATA

cat > $VM_AGENT_DATA/scrape.yml <<'EOF'
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: vmstorage
    static_configs:
      - targets:
          - 10.0.0.1:8482
          - 10.0.0.2:8482
          - 10.0.0.3:8482
        labels:
          cluster: vm_cluster_3n
          component: vmstorage

  - job_name: vminsert
    static_configs:
      - targets:
          - 10.0.0.1:8480
          - 10.0.0.2:8480
          - 10.0.0.3:8480
        labels:
          cluster: vm_cluster_3n
          component: vminsert

  - job_name: vmselect
    static_configs:
      - targets:
          - 10.0.0.1:8481
          - 10.0.0.2:8481
          - 10.0.0.3:8481
        labels:
          cluster: vm_cluster_3n
          component: vmselect

  - job_name: vmagent
    static_configs:
      - targets:
          - 127.0.0.1:8429
        labels:
          cluster: vm_cluster_3n
          component: vmagent

  - job_name: node-exporter
    static_configs:
      - targets:
          - 10.0.0.1:9100
          - 10.0.0.2:9100
          - 10.0.0.3:9100
EOF

For the single-node edition, change scrape_configs to a vmsingle job (127.0.0.1:8428) and drop vmstorage / vminsert / vmselect.

Restart vmagent after the change:

1
$CONTAINER_CLI restart $VM_AGENT_NAME

4.2 Verify Scraping

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# vmagent target status (State should be up)
curl -s http://127.0.0.1:8429/targets | grep -E 'vmstorage|vminsert|vmselect|vmagent'

# Sample metrics from each component
curl -s http://10.0.0.1:8482/metrics | grep -E '^vm_rows|^vm_active_merges' | head
curl -s http://10.0.0.1:8480/metrics | grep vm_rpc_vmstorage_is_reachable
curl -s http://10.0.0.1:8481/metrics | grep vm_cache_requests_total | head

# Check through vmselect whether the self-monitoring metrics made it into storage
curl -s 'http://10.0.0.1:8481/select/0/prometheus/api/v1/query?query=up{component="vmstorage"}'

Key metrics to watch:

Metric	Meaning
`vm_rows`	Current number of time-series rows in vmstorage
`vm_rpc_vmstorage_is_reachable`	Whether vminsert/vmselect can reach vmstorage (1 = healthy)
`vmagent_remotewrite_rows_sent_total`	Number of rows vmagent wrote successfully
`vm_http_requests_total`	HTTP request volume per component
`process_resident_memory_bytes`	Process memory usage

4.3 Import Grafana Dashboards

Use the Prometheus-type data source configured in section 5.1 (pointing at vmselect).

In Grafana, go to Dashboards → New → Import on the left and enter a dashboard ID to import:

Dashboard	ID	Use	Description
VictoriaMetrics - cluster	11176	Cluster edition	vmstorage / vminsert / vmselect RPC, cache, resources
VictoriaMetrics - vmagent	12683	vmagent	Scrape targets, remote_write rate and errors
VictoriaMetrics - single-node	10229	Single-node edition	vmsingle writes, queries, storage, resources

Import steps:

Enter the ID (e.g. 11176) → Load
For the VictoriaMetrics data source, pick the one created in section 5.1 (for a cluster, enter the vmselect URL)
Import

For the cluster edition, import both 11176 (components) and 12683 (vmagent); for the single-node edition, import 10229.

In the dashboard, filter by the instance or component label to view the vmstorage status of each of the three nodes separately.

5. Integrating with Grafana / Prometheus

5.1 Grafana Data Source

Deployment Mode	URL
Single-node	`http://<vm-ip>:8428`
Cluster	`http://<vmselect-ip>:8481/select/0/prometheus/`

Choose Prometheus as the type; it is fine once Save & Test passes.

5.2 remote_write Ingestion

Single-node edition:

1
2
remote_write:
  - url: http://10.0.0.1:8428/api/v1/write

Cluster edition:

1
2
remote_write:
  - url: http://10.0.0.1:8480/insert/0/prometheus/api/v1/write