Open-Source GPU Virtualization Projects on Kubernetes

1. k8s-device-plugin

https://github.com/NVIDIA/k8s-device-plugin is the official Kubernetes device plugin from NVIDIA, used to manage and allocate NVIDIA GPU resources in a Kubernetes cluster.

By interacting with kubelet, k8s-device-plugin automatically discovers and registers GPU devices and exposes them to the Kubernetes scheduler as resources. It supports many GPU models and is able to handle GPU partitioning and sharing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  restartPolicy: Never
  containers:
    - name: cuda-container
      image: nvidia/cuda:11.8.0-base
      resources:
        limits:
          nvidia.com/gpu: 1 # use one GPU
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

2. gpu-operator

https://github.com/NVIDIA/gpu-operator is an official NVIDIA component that provides full GPU lifecycle management: driver management, container runtime integration, device discovery, node feature discovery, and more.

gpu-operator not only integrates k8s-device-plugin, but also provides the following GPU resource allocation methods:

2.1 MIG

Multi-Instance GPU, or MIG for short, is a technology provided by NVIDIA.

MIG can split a single GPU from the Blackwell, Hopper, or Ampere architecture into multiple independent instances, where each instance has its own compute, memory, and bandwidth. For specific models, refer to https://docs.nvidia.com/datacenter/tesla/mig-user-guide/

It is worth noting that MIG is a hardware-level partitioning technology, so only GPUs that support MIG can use it.

To use it on Kubernetes, you must first install the k8s-device-plugin and gpu-feature-discovery components.

First, configure MIG:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-mig-config
  namespace: nvidia-device-plugin
data:
  config.json: |
    {
      "mig-configs": [
        {
          "devices": ["all"],
          "mig-enabled": true,
          "mig-devices": [
            {"profile": "1g.5gb", "count": 7}
          ]
        }
      ]
    }    

1g.5gb is a MIG profile, meaning each instance has 1 GPU slice (a single card is divided into several slices) and 5 GB of memory.

Use MIG resources in a Pod like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
apiVersion: v1
kind: Pod
metadata:
  name: mig-demo
spec:
  restartPolicy: Never
  containers:
    - name: demo
      image: nvidia/cuda:11.8.0-base
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/mig-1g.5gb: 1 # use one MIG instance

2.2 Time-Slicing

Almost all NVIDIA GPUs support Time-Slicing. Time-Slicing allows a physical GPU to be split into multiple virtual devices, each of which can be assigned to a different Pod. Each virtual device shares the physical GPU’s compute resources in time.

Time-Slicing is only GPU sharing at the scheduling level and does not provide isolation at the resource level, so resource contention is a problem.

First, configure Time-Slicing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
data:
  default: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu  # resource name
          replicas: 4           # split each physical GPU into 4 virtual devices    

Use Time-Slicing resources in a Pod like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
apiVersion: v1
kind: Pod
metadata:
  name: gpu-timeslicing-demo
spec:
  containers:
    - name: cuda-container
      image: nvidia/cuda:11.8.0-base
      command: ["bash", "-c", "nvidia-smi && sleep 3600"]
      resources:
        limits:
          nvidia.com/gpu: 1 # use one GPU, which may actually share the same physical GPU with three other Pods

2.3 MPS

MPS (Multi-Process Service) is a technology provided by NVIDIA for running multiple CUDA applications on the same physical GPU. MPS allows multiple processes to share the compute resources of a single GPU, thereby improving GPU utilization.

Compared with Time-Slicing, MPS provides better resource isolation and performance optimization. MPS manages the execution of multiple CUDA applications by creating an MPS Server process.

First, configure MPS:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: mps-policy
spec:
  devicePlugin:
    enabled: true
    env:
      - name: NVIDIA_MPS_ENABLE
        value: "1"

Use MPS resources in a Pod:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
apiVersion: v1
kind: Pod
metadata:
  name: mps-demo
spec:
  containers:
    - name: mps-container
      image: nvidia/cuda:11.8.0-base
      env:
        - name: NVIDIA_MPS_ENABLE
          value: "1"
      command: ["/bin/bash", "-c", "nvidia-smi && sleep 3600"]
      resources:
        limits:
          nvidia.com/gpu: 1

3. gpu-manager

https://github.com/tkestack/gpu-manager uses vCUDA technology. vCUDA (virtual CUDA) is an API Forwarding/Remoting implementation of a vGPU.

vCUDA intercepts and redirects CUDA API calls at the user layer, and builds a logical image of the pGPU inside a VM — namely the vGPU — to achieve fine-grained partitioning, recombining, and reuse of GPU resources, supporting advanced VM features such as multi-machine concurrency and suspend/resume.

For allocations of one or more whole cards, gpu-manager controls which GPUs are visible to the application inside the container through the environment variable NVIDIA_VISIBLE_DEVICES, and mounts only the visible devices into the container.

For allocations of less than one card, gpu-manager injects LD_LIBRARY_PATH and, with the help of https://github.com/tkestack/vcuda-controller, hijacks CUDA and NVML library function calls and modifies their return values to partition GPU resources.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: v1
kind: Pod
metadata:
  name: vcuda
spec:
  restartPolicy: Never
  containers:
    - image: nvidia/cuda:11.8.0-base
      name: nvidia
      command:
        - /usr/local/nvidia/bin/nvidia-smi
        - pmon
        - -d
        - 10
      resources:
        requests:
          tencent.com/vcuda-core: 50
          tencent.com/vcuda-memory: 30
        limits:
          tencent.com/vcuda-core: 50
          tencent.com/vcuda-memory: 30

Here we allocate 7680 MiB of GPU memory and 0.5 GPU. The unit of tencent.com/vcuda-core is 0.01 GPU core, and the unit of tencent.com/vcuda-memory is 256 MiB of GPU memory.

4. HAMi

https://github.com/Project-HAMi/HAMi uses the vCUDA approach, partitioning and sharing GPU resources by hijacking CUDA API calls.

Because they belong to the same vCUDA technology stack, HAMi and gpu-manager are used in a similar way. The difference is that the gpu-manager project is no longer maintained, while HAMi is still actively updated.

HAMi integrates not only NVIDIA’s MIG and Time-Slicing, but can also manage other AI accelerator cards such as NPU, MLU, and DCU. It is also optimized at the scheduling level, able to allocate resources according to custom policies.

Below is an example Pod configuration using HAMi:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: Pod
metadata:
  name: mps-demo
spec:
  containers:
    - name: mps-container
      image: nvidia/cuda:11.8.0-base
      env:
        - name: NVIDIA_MPS_ENABLE
          value: "1"
      command: ["/bin/bash", "-c", "nvidia-smi && sleep 3600"]
      resources:
        limits:
          nvidia.com/gpu: 1 # use 1 GPU
          nvidia.com/gpucores: 30 # use 30% of each GPU's cores
          nvidia.com/gpumem: 3000 # request 3G of GPU memory