1. What Is nvidia-smi
nvidia-smi stands for NVIDIA System Management Interface, the interface NVIDIA provides for managing and monitoring GPUs.
nvidia-smi calls NVML. NVML stands for NVIDIA Management Library, a library that provides a set of C APIs for NVIDIA GPU monitoring and management.
1.1 Queryable State
- ECC error counts
- GPU utilization
- Active compute processes
- Clocks and PState
- Temperature and fan speed
- Power management
- Hardware identification
1.2 Modifiable State
- ECC mode
- ECC reset
- Compute mode
- Persistence mode
2. nvidia-smi Field Meanings
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
| nvidia-smi
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A100-SXM4-80GB On | 00000000:27:00.0 Off | 0 |
| N/A 32C P0 65W / 400W | 4MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A100-SXM4-80GB On | 00000000:2A:00.0 Off | 0 |
| N/A 29C P0 63W / 400W | 4MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA A100-SXM4-80GB On | 00000000:51:00.0 Off | 0 |
| N/A 31C P0 74W / 400W | 34221MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 3 NVIDIA A100-SXM4-80GB On | 00000000:57:00.0 Off | 0 |
| N/A 33C P0 66W / 400W | 4MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 4 NVIDIA A100-SXM4-80GB On | 00000000:9E:00.0 Off | 0 |
| N/A 31C P0 60W / 400W | 4MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 5 NVIDIA A100-SXM4-80GB On | 00000000:A4:00.0 Off | 0 |
| N/A 29C P0 62W / 400W | 4MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 6 NVIDIA A100-SXM4-80GB On | 00000000:C7:00.0 Off | 0 |
| N/A 28C P0 64W / 400W | 4MiB / 81920MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 7 NVIDIA A100-SXM4-80GB On | 00000000:CA:00.0 Off | 0 |
| N/A 33C P0 92W / 400W | 7MiB / 81920MiB | 72% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 2 N/A N/A 3085965 C /home/xxx/.conda/envs/chat/bin/python 34208MiB |
+---------------------------------------------------------------------------------------+
|
| Field | Description |
|---|
| NVIDIA-SMI | Version number of nvidia-smi |
| Driver Version | Driver version number |
| CUDA Version | CUDA version number |
| GPU | GPU card index |
| GPU Name | GPU name and memory capacity |
| Persistence-M | Whether persistence mode is enabled. On means enabled, Off means disabled. When enabled, the GPU stays in its highest performance state |
| Bus-Id | PCIe bus address the GPU sits on |
| Disp.A | Whether a display is connected to the GPU output port. On means connected, Off means not connected |
| Volatile Uncorr. ECC | Volatile ECC memory error count for uncorrected errors. Used to detect memory errors |
| Fan | Fan speed, N/A means there is no fan or no fan speed reading |
| Temp | GPU temperature |
| Perf | Performance state. P0 is the maximum performance state, P8 is the minimum performance state |
| Pwr | Usage/Cap: current power draw and power cap |
| Memory-Usage | Used GPU memory / total GPU memory |
| GPU-Util | GPU utilization |
| Compute M. | Compute mode. Default is the default mode |
| MIG M. | MIG (Multi-Instance GPU) mode, which splits one physical GPU into multiple independent, isolated instances. Disabled means it is not enabled |
| Field | Description |
|---|
| GPU | ID of the GPU device |
| GI | Global ID, for multi-GPU systems, a unified ID for all cuda contexts of one process |
| CI | Compute Instance ID, within the same GPU process, the ID that distinguishes different cuda contexts |
| PID | Process ID |
| Type | Process type, C means a CUDA process, G means a Graphics process |
| Process name | Process name |
| GPU Memory Usage | Amount of memory this process currently occupies on the GPU |
3. Common Parameters
nvidia-smi -l refreshes the status periodically
Refresh every 5 seconds
nvidia-smi -L shows the graphics card model
1
2
3
4
5
| nvidia-smi -L
GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-x-8bff-5236-2111-x)
GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-x-2a64-20a8-8c5b-x)
...
|
nvidia-smi -q shows detailed GPU status
You can use the -i parameter to specify the GPU index; if not specified, it queries all by default.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
| nvidia-smi -q -i 0
==============NVSMI LOG==============
Driver Version : 535.129.03
CUDA Version : 12.2
Attached GPUs : 8
GPU 00000000:27:00.0
Product Name : NVIDIA A100-SXM4-80GB
Product Brand : NVIDIA
Product Architecture : Ampere
Display Mode : Enabled
Display Active : Disabled
Persistence Mode : Enabled
Addressing Mode : None
MIG Mode
Current : Disabled
Pending : Disabled
|
nvidia-smi -q -x outputs the queried information in XML format
1
2
3
4
5
6
7
8
9
| nvidia-smi -q -i 0 -x
<?xml version="1.0" ?>
<!DOCTYPE nvidia_smi_log SYSTEM "nvsmi_device_v12.dtd">
<nvidia_smi_log>
<driver_version>535.129.03</driver_version>
<cuda_version>12.2</cuda_version>
<attached_gpus>8</attached_gpus>
...
</nvidia_smi_log>
|
nvidia-smi --query-gpu=gpu_name --format=csv queries specific fields
The --query-gpu parameter specifies the information to query, and the --format parameter specifies the output format.
1
2
3
4
5
6
7
8
9
10
| nvidia-smi --query-gpu=temperature.gpu,utilization.gpu,utilization.memory,memory.total --format=csv,noheader
32, 0 %, 0 %, 81920 MiB
30, 0 %, 0 %, 81920 MiB
31, 0 %, 0 %, 81920 MiB
49, 72 %, 47 %, 81920 MiB
31, 0 %, 0 %, 81920 MiB
28, 0 %, 0 %, 81920 MiB
28, 0 %, 0 %, 81920 MiB
30, 0 %, 0 %, 81920 MiB
|
nvidia-smi -q -d SUPPORTED_CLOCKS shows the GPU clock frequencies
Use the -i parameter to specify the GPU index; if not specified, it queries the clock frequencies of all GPUs by default.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
| nvidia-smi -q -d SUPPORTED_CLOCKS -i 0
==============NVSMI LOG==============
Timestamp : Thu Feb 1 14:33:03 2024
Driver Version : 535.129.03
CUDA Version : 12.2
Attached GPUs : 8
GPU 00000000:27:00.0
Supported Clocks
Memory : 1593 MHz
Graphics : 1410 MHz
Graphics : 1395 MHz
Graphics : 1380 MHz
Graphics : 1365 MHz
Graphics : 1350 MHz
Graphics : 1335 MHz
Graphics : 1320 MHz
|
4. Common Subcommands
nvidia-smi nvlink -s shows the NVLink network status
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| nvidia-smi nvlink -s
GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-d604695a-8bff-5236-2111-59cae59c2a48)
Link 0: 25 GB/s
Link 1: 25 GB/s
Link 2: 25 GB/s
Link 3: 25 GB/s
Link 4: 25 GB/s
Link 5: 25 GB/s
Link 6: 25 GB/s
Link 7: 25 GB/s
Link 8: 25 GB/s
Link 9: 25 GB/s
Link 10: 25 GB/s
Link 11: 25 GB/s
|
nvidia-smi topo -m shows the connection topology
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
| nvidia-smi topo -m
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV12 NV12 NV12 NV12 NV12 NV12 NV12 SYS 0-31,64-95 0 N/A
GPU1 NV12 X NV12 NV12 NV12 NV12 NV12 NV12 SYS 0-31,64-95 0 N/A
GPU2 NV12 NV12 X NV12 NV12 NV12 NV12 NV12 SYS 0-31,64-95 0 N/A
GPU3 NV12 NV12 NV12 X NV12 NV12 NV12 NV12 SYS 0-31,64-95 0 N/A
GPU4 NV12 NV12 NV12 NV12 X NV12 NV12 NV12 SYS 32-63,96-127 1 N/A
GPU5 NV12 NV12 NV12 NV12 NV12 X NV12 NV12 SYS 32-63,96-127 1 N/A
GPU6 NV12 NV12 NV12 NV12 NV12 NV12 X NV12 SYS 32-63,96-127 1 N/A
GPU7 NV12 NV12 NV12 NV12 NV12 NV12 NV12 X SYS 32-63,96-127 1 N/A
NIC0 SYS SYS SYS SYS SYS SYS SYS SYS X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_bond_0
|
NV12 means there are 12 NVLinks; at 25 GB/s each, the interconnect speed between GPUs here reaches 300 GB/s.
5. Common Configuration Commands
- Enable persistence mode (now superseded by the nvidia-persistenced daemon)
You can use the -i parameter to specify the GPU index.
Persistence mode means that the driver stays loaded even when no application is using the GPU.
Persistence mode draws more power, but for short-lived tasks it shortens the startup latency of GPU programs.
- Enable ECC mode, takes effect after a reboot
You can use the -i parameter to specify the GPU index.
To disable it, use -e 0, which requires a reboot to take effect.
Note that after enabling ECC, although memory errors can be avoided, you lose 15-25% of performance, and the available GPU memory is reduced as well.
There are three compute modes in total: 0/Default, where multiple processes share the GPU and there is contention and waiting; 2/Prohibited, which disables the GPU; 3/Exclusive, where a process has exclusive access
nvidia-smi -c 0
6. Common Exception Handling
6.1 nvidia-smi Fails Inside a Container
1
| CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 36: API call is not supported in the installed CUDA driver
|
Set the environment variable
1
| LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu:/usr/local/nvidia/lib
|
The cuda version in the container is relatively old, so cuda.so does not match the driver; the LD_LIBRARY_PATH environment variable above is used to bypass the old version of the driver.
6.2 Disabling a Graphics Card
- Check the card’s PCI location
1
2
3
4
| nvidia-smi --query-gpu=index,pci.bus_id --format=csv
index, pci.bus_id
0, 00000000:18:00.0
|
The first column is the card index, the second column is the card’s PCI location.
- Disable the specified GPU card
1
| nvidia-smi drain -p 0000:18:00.0 -m 1
|
-m 1 means drain state, -m 0 means off drain state. After the command finishes, the card is invisible to nvidia-smi but still visible in lspci.
- Check the card’s drain state
1
| nvidia-smi drain -p 0000:18:00.0 -q
|
