NVIDIA_VISIBLE_DEVICES

Specifies the GPU devices visible to the program

1

CUDA_VISIBLE_DEVICES=0,1

Available values:

• 1,2, a comma-separated list of GPU UUIDs or indices
• all, all GPUs
• none, load the driver but with no GPU access
• void, do not load the driver

NVIDIA_DRIVER_CAPABILITIES

Controls which driver libraries/binaries will be installed inside the container

1

NVIDIA_DRIVER_CAPABILITIES=compute,utility

Available values:

• compute, required by CUDA and OpenCL applications.
• compat32, required to run 32-bit applications.
• graphics, required to run OpenGL and Vulkan applications.
• utility, required by nvidia-smi and NVML.
• video, required to use the video codec SDK.
• display, required to take advantage of X11 display.

NVIDIA_REQUIRE_CUDA

Used to specify the required CUDA version and driver version

1

NVIDIA_REQUIRE_CUDA="cuda>=11.0 driver>=450"

1

NVIDIA_REQUIRE_CUDA="cuda>=11.7"

NVIDIA_REQUIRE_DRIVER

Used to specify the required driver version

1

NVIDIA_REQUIRE_DRIVER="driver>=470"

NVIDIA_REQUIRE_BRAND

Used to specify the required GPU brand

1
2

NVIDIA_REQUIRE_BRAND="tesla"
NVIDIA_REQUIRE_BRAND="geforce"

NVIDIA_REQUIRE_ARCH

Used to specify the required GPU architecture

1

NVIDIA_REQUIRE_ARCH "maxwell pascal volta turing ampere"

CUDA_DEVICE_ORDER

Orders GPU devices starting from 0 in the order of PCI_BUS_ID.

1

export CUDA_VISIBLE_DEVICES='PCI_BUS_ID'

Use only the first two GPUs

LD_LIBRARY_PATH

Specifies the path to dynamic link libraries, ensuring that CUDA library files can be found at runtime.

1

LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/local/nvidia/lib

CUDNN_PATH

Specifies the installation path of the cuDNN library

1

CUDNN_PATH=/usr/local/cuda

CUDA_HOME

The path searched for CUDA library files and tools when building CUDA applications

1

CUDA_HOME=/usr/local/cuda

CUDA_PATH

The path where NVIDIA’s own development tools search for CUDA library files and tools

1

CUDA_PATH=/usr/local/cuda

NCCL_IB_HCA

Used to specify the RDMA interface to be used for communication

1

NCCL_IB_HCA=mlx5_0,mlx5_1

NCCL_PXN_DISABLE

Disables inter-node communication that uses a non-local NIC

1

NCCL_PXN_DISABLE=1

NCCL_IB_TIMEOUT

Controls the InfiniBand timeout, with a value range of 1-22

1

NCCL_IB_TIMEOUT=22

Set a larger value to avoid NCCL timeouts

NCCL_IB_RETRY_CNT

Controls the number of InfiniBand retries

1

NCCL_IB_RETRY_CNT=13

NCCL_DEBUG

Enables NCCL debug logging

1

NCCL_DEBUG=INFO

NCCL_DEBUG_FILE

Specifies the path of the NCCL debug log file, making debugging easier

1

NCCL_DEBUG_FILE=/tmp/nccl.log

NCCL_IB_PCI_RELAXED_ORDERING

Can improve the performance of InfiniBand networks in virtualized environments

1

NCCL_IB_PCI_RELAXED_ORDERING=1

0 means disabled, 1 means enabled, 2 means automatic; the default is 2.

NCCL_SOCKET_IFNAME

Specifies the network interface used for communication

1

NCCL_SOCKET_IFNAME=eth0

NCCL_IB_DISABLE

Disables the InfiniBand network

1

NCCL_IB_DISABLE=1

NCCL_IB_GID_INDEX

Specifies the GID index to use

1

NCCL_IB_GID_INDEX=3

0 and 1 mean ipv5 is used; 2 and 3 mean ipv4.

Among these, 2 means the RoCE v1 protocol, and 3 means the RoCE v2 protocol.