This page looks best with JavaScript enabled

Volcano Basics

 ·  ☕ 9 min read

1. Introduction to Volcano

Volcano is a Kubernetes-based resource scheduling system open-sourced by Huawei. Compared with the native scheduler, its notable features are:

  • Supports gang scheduling

Scheduling batch jobs easily runs into deadlock: for example, two jobs each need 10 Pods running simultaneously to start. When both jobs are submitted at the same time, it is possible that only part of the Pods of each are scheduled, so neither job can run properly and they wait on each other. gang scheduling exists to solve exactly this problem.

  • Scheduling queues

Configuring different scheduling queues makes it possible to preempt resources, control quotas, and so on.

  • Hardware awareness

Awareness of hardware resources such as NUMA and GPU lets Pods use hardware resources more efficiently.

Volcano extends and optimizes on top of Kubernetes’ native scheduling capabilities, so it also supports basic nodeSelector, nodeAffinity, and the like. It also supports Extended Resource, which matters a great deal for scheduling-level awareness of resources such as GPUs and IB NICs.

2. Installation

  • Add the Helm repo
1

helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
  • Install a specific version
1

helm install volcano volcano-sh/volcano -n volcano-system --create-namespace --version 1.8.2

1
2
3
4
5
6
7
8
9

kubectl get crd |grep volcano

commands.bus.volcano.sh              2024-03-21T03:41:33Z
jobflows.flow.volcano.sh             2024-03-21T03:41:33Z
jobs.batch.volcano.sh                2024-03-21T03:41:33Z
jobtemplates.flow.volcano.sh         2024-03-21T03:41:33Z
numatopologies.nodeinfo.volcano.sh   2024-03-21T03:41:33Z
podgroups.scheduling.volcano.sh      2024-03-21T03:41:33Z
queues.scheduling.volcano.sh         2024-03-21T03:41:33Z
  • commands.bus.volcano.sh

Used to interact with the Volcano system. It lets users trigger specific operations by creating Command objects, such as pausing/resuming jobs, rescheduling, and so on.

  • jobflows.flow.volcano.sh

The JobFlow object describes execution dependencies among multiple jobs.

A common use case is a data processing pipeline: with Jobflow, job execution order can be automatically ordered correctly according to the dependencies.

  • jobs.batch.volcano.sh

Job is Volcano’s most central resource object, used to submit and run batch jobs. It supports several workload types, such as a single Job, a Job array, and periodic jobs.

A Job can also mount data volumes, configure resource requirements, and so on.

  • jobtemplates.flow.volcano.sh

JobTemplate provides a template mechanism for creating similar jobs. Users need only define a job spec once, and can then quickly and conveniently create many instances from the template.

It is especially suited to scenarios that need to run dozens or hundreds of similar jobs at once, greatly reducing management and operations costs.

  • numatopologies.nodeinfo.volcano.sh

The NumaTopology object describes a node’s NUMA information.

  • podgroups.scheduling.volcano.sh

The PodGroup object schedules multiple Pods as a single unit.

When multiple Pods need to run at the same time, you can use the PodGroup object.

  • queues.scheduling.volcano.sh

The Queue object defines job queues to achieve resource isolation and fair scheduling.

In multi-tenant scenarios, different teams or departments can create their own queues as needed and set resource quotas and priority parameters for them. Queues prevent mutual interference and enable predictable, controllable resource allocation.

4. Job Plugins Customize Pod Execution

4.1 Three Commonly Used Plugins

Volcano ships with some built-in plugins. To develop a custom plugin, implement the PluginInterface according to the source at https://github.com/volcano-sh/volcano/tree/master/pkg/controllers/job/plugins. Here is an example:

1
2
3
4
5
6
7
8
9

apiVersion: batch.volcano.sh/v1beta1
kind: Job
metadata:
  name: my-job
spec:
  plugins:
    ssh: []
    env: []
    svc: []

These plugins can satisfy some customization needs:

  • ssh plugin

Configures SSH mutual trust between Pods and provides passwordless login

  • svc plugin

Provides the network information a job needs to run, such as the hosts file and a headless service, enabling automated configuration of compute cluster parameters

  • env plugin

Provides the environment variables a job needs to run

4.2 Injection Mechanism

  • The env plugin injects VK_TASK_INDEX and VC_TASK_INDEX.

Injection mechanism:

1
2
3
4
5
6
7

spec:
  containers:
    - env:
        - name: VK_TASK_INDEX
          value: "0"
        - name: VC_TASK_INDEX
          value: "0"

The plugin derives the index value from the Pod name and sets it directly into env.

  • The svc plugin injects VC_DEMO_NUM and VC_DEMO_HOSTS

Injection mechanism:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

spec:
  containers:
    - env:
        - name: VC_DEMO_HOSTS
          valueFrom:
            configMapKeyRef:
              key: VC_DEMO_HOSTS
              name: my-plugins-job-svc
        - name: VC_DEMO_NUM
          valueFrom:
            configMapKeyRef:
              key: VC_DEMO_NUM
              name: my-plugins-job-svc

1
2
3
4

kubectl get cm my-plugins-job-svc

NAME                 DATA   AGE
my-plugins-job-svc   3      15m

The values of VC_DEMO_NUM and VC_DEMO_HOSTS are stored in my-plugins-job-svc.

  • The ssh plugin injects the public and private keys for ssh access

Injection mechanism:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

spec:
  containers:
    volumes:
      - name: my-plugins-job-ssh
        secret:
          defaultMode: 384
          items:
            - key: id_rsa
              path: .ssh/id_rsa
            - key: id_rsa.pub
              path: .ssh/id_rsa.pub
            - key: authorized_keys
              path: .ssh/authorized_keys
            - key: config
              path: .ssh/config
          secretName: my-plugins-job-ssh

1
2
3
4

kubectl get secret my-plugins-job-ssh

NAME                 TYPE     DATA   AGE
my-plugins-job-ssh   Opaque   4      22m

my-plugins-job-ssh stores the ssh public and private keys, and Volcano mounts the keys into the Pod.

4.3 Testing the Plugins

  • Create a Job in which multiple Pods run simultaneously
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

cat <<EOF | kubectl apply -f -
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: my-plugins-job
spec:
  minAvailable: 3
  plugins:
    ssh: []
    env: []
    svc: []
  tasks:
    - replicas: 3
      name: demo
      template:
        spec:
          containers:
            - name: demo
              image: shaowenchen/demo:sshd
EOF

This creates three Pods running simultaneously.

  • Check how the Pods are running
1
2
3
4
5
6

kubectl get pod -o wide

NAME                            READY   STATUS    RESTARTS   AGE    IP
my-plugins-job-demo-0           1/1     Running   0          79s    10.244.228.250
my-plugins-job-demo-1           1/1     Running   0          79s    10.244.8.240
my-plugins-job-demo-2           1/1     Running   0          79s    10.244.228.249
  • Check the injected environment variables

Enter the container

1

kubectl exec -it my-plugins-job-demo-0 -- bash

Print the environment variables Volcano injected

1
2
3
4
5
6

env | grep -E '^VC_|^VK_'

VC_DEMO_NUM=3
VK_TASK_INDEX=0
VC_TASK_INDEX=0
VC_DEMO_HOSTS=my-plugins-job-demo-0.my-plugins-job,my-plugins-job-demo-1.my-plugins-job,my-plugins-job-demo-2.my-plugins-job

VC_DEMO_NUM is the total number of tasks; in the source, VK_TASK_INDEX and VC_TASK_INDEX are assigned equal values and both represent the Task’s index, which differs in every Pod; VC_DEMO_HOSTS is the IP list of the current task.

  • Test the passwordless ssh plugin

Enter the container

1

kubectl exec -it my-plugins-job-demo-0 -- bash

Passwordless ssh to another Pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

ssh 10.244.8.240
Warning: Permanently added '10.244.8.240' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-144-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/pro

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Note that here you can only access by IP, not by Pod Name, because Volcano does not write other Pods’ IPs and Names into /etc/hosts.

5. Configuring a Deployment to Use Volcano to Control Resource Usage

Here is an example that limits a Deployment to at most 2 CPU cores.

  • Create the queue
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

cat <<EOF | kubectl apply -f -
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: my-node-queue
spec:
  weight: 1
  reclaimable: false
  capability:
    cpu: 2
EOF

This creates a queue with only 2 CPU cores, bound to the node group my-node-group.

Here weight is the relative share of cluster resources, and it is a soft constraint; reclaimable says whether the queue may be reclaimed, as determined by weight; capability is the queue’s resource limit.

  • Create the Deployment
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ubuntu-with-volcano
  labels:
    app: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      schedulerName: volcano
      containers:
        - name: demo
          image: shaowenchen/demo:ubuntu
          resources:
            requests:
              cpu: 1
EOF

Setting schedulerName to volcano means the Volcano scheduler is used.

  • Check the Pods
1
2
3
4

kubectl get pods -l app=demo

NAME                                  READY   STATUS    RESTARTS   AGE
ubuntu-with-volcano-97c94f9fb-bfgrh   1/1     Running   0          6m24s
  • Scale up the Deployment
1

kubectl scale deployment/ubuntu-with-volcano --replicas=3

At this point only two of the three Pods are Running, because Volcano limits the Deployment to at most 2c CPU.

1
2
3
4
5
6

kubectl get pods -l app=demo

NAME                                  READY   STATUS    RESTARTS   AGE
ubuntu-with-volcano-97c94f9fb-25nb7   1/1     Running   0          27s
ubuntu-with-volcano-97c94f9fb-6fd64   0/1     Pending   0          27s
ubuntu-with-volcano-97c94f9fb-bfgrh   1/1     Running   0          7m31s

6. Configuring a Job to Use Volcano to Throttle Concurrent Execution

This creates a Job that requires at least 3 Pods to run together.

You can also achieve this with a plain Kubernetes batch/v1 Job by configuring completions and parallelism. But the Queue Volcano provides can control resource usage, and Policy can control Task lifecycle policies, giving more precise control over Job execution.

  • Create the Job
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

cat <<EOF | kubectl apply -f -
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: my-job
spec:
  minAvailable: 3
  schedulerName: volcano
  queue: default
  policies:
    - event: PodEvicted
      action: RestartJob
  tasks:
    - replicas: 30
      name: demo
      policies:
      - event: TaskCompleted
        action: CompleteJob
      template:
        spec:
          containers:
            - image: ubuntu
              name: demo
              command: ["sleep", "5"]
              resources:
                requests:
                  cpu: 20
          restartPolicy: Never
EOF

Where:

1
2
3

policies:
  - event: PodEvicted
    action: RestartJob

means that if a Pod is evicted, the Job is restarted.

1
2
3

policies:
  - event: TaskCompleted
    action: CompleteJob

means that if a Task completes, the Job is completed.

Through Event and Action, you can control a Job’s state and behavior.

  • Check how the Pods were created
1
2
3
4
5
6
7
8
9

kubectl get pod

NAME                            READY   STATUS    RESTARTS   AGE
my-job-demo-0                   1/1     Running   0          7s
my-job-demo-1                   1/1     Running   0          7s
my-job-demo-10                  0/1     Pending   0          7s
...
my-job-demo-2                   1/1     Running   0          7s
...

Because I set the Pods’ CPU request to 20 and the cluster does not have enough resources, only 3 of the 30 Pods can run at a time.

After execution finishes, the Pods are not deleted but remain in the Completed state. Since a Pod’s ownerReferences is the Job, deleting the Job also deletes the Pods.

微信公众号
WRITTEN BY
微信公众号

See Also