This page looks best with JavaScript enabled

How to Enable GDS on a GPU Host

 ·  ☕ 8 min read

1. What GDS (GPUDirectStorage) Is

GDS lets an RDMA NIC access GPU memory directly, which helps increase the IO bandwidth for GPU applications reading and writing files, reduces IO latency, and lowers their CPU load.

Once the client has GDS enabled, files are opened in O_DIRECT mode and the client no longer caches file data. When the application layer reads or writes a file, the client uses nvidia-fs.ko to translate the application’s GPU memory address into a DMA address and sends it to the server, where the server’s RDMA NIC is responsible for reading from or filling the client’s DMA address.

2. Requirements

2.1 System Requirements

  • OS

Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04, RHEL 8, RHEL 9

  • CUDA Toolkit

  • GCC

2.2 Storage Requirements

When using GDS, the application layer must call the cuFile API for it to work properly. Only a limited number of storage systems have implemented support for GDS:

  • NVMe
  • NVMeOF
  • SCSI
  • ScaleFlux CSD
  • NVMesh
  • DDN EXAScaler
  • IBM Spectrum Scale
  • NFS
  • BeeGFS
  • WekaFS
  • …

Here is a support list from late 2021 you can refer to: https://developer.nvidia.com/zh-cn/blog/accelerating-io-in-the-modern-data-center-magnum-io-storage-partnerships/

3. Installing GDS

3.1 Installing Dependencies

  • Install cuda-keyring
1
2
3
4
5

export distro=ubuntu2004
export arch=x86_64
wget https://developer.download.nvidia.com/compute/cuda/repos/$distro/$arch/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
apt-get update
  • Install nvidia-fs
1
2
3

apt list --installed | grep nvidia-fs
apt-cache search nvidia-fs
apt-get install nvidia-fs-dkms nvidia-fs
  • Check the current Driver version
1
2
3

nvidia-smi | grep "Driver Version"

| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
  • Install libnvidia-nscq

Before installing, check whether it is already installed, and then look for the version that matches your Driver version.

1
2
3

apt list --installed | grep libnvidia_nscq
apt-cache search libnvidia-nscq
apt-get install libnvidia-nscq-535
  • Install fabricmanager

Before installing, check whether it is already installed, and then look for the version that matches your Driver version.

1
2
3

apt list --installed | grep fabricmanager
apt-cache search fabricmanager
apt-get install nvidia-fabricmanager-535

3.2 Installing GDS

  • Install GDS
1
2
3

apt list --installed | grep nvidia-gds
apt-cache search nvidia-gds
apt-get install nvidia-gds-12-2
  • Check GDS status
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

/usr/local/cuda-12.2/gds/tools/gdscheck.py -p

 GDS release version: 1.7.2.10
 nvidia_fs version:  2.22 libcufile version: 2.12
 Platform: x86_64
 ============
 ENVIRONMENT:
 ============
 =====================
 DRIVER CONFIGURATION:
 =====================
 NVMe               : Supported
 NVMeOF             : Unsupported
 SCSI               : Unsupported
 ScaleFlux CSD      : Unsupported
 NVMesh             : Unsupported
 DDN EXAScaler      : Unsupported
 IBM Spectrum Scale : Unsupported
 NFS                : Unsupported
 BeeGFS             : Unsupported
 WekaFS             : Unsupported
 Userspace RDMA     : Unsupported
 --Mellanox PeerDirect : Enabled
 --rdma library        : Not Loaded (libcufile_rdma.so)
 --rdma devices        : Not configured
 --rdma_device_status  : Up: 0 Down: 0
 =====================
 CUFILE CONFIGURATION:
 =====================
 properties.use_compat_mode : true
 properties.force_compat_mode : false
 properties.gds_rdma_write_support : true
 properties.use_poll_mode : false
 properties.poll_mode_max_size_kb : 4
 properties.max_batch_io_size : 128
 properties.max_batch_io_timeout_msecs : 5
 properties.max_direct_io_size_kb : 16384
 properties.max_device_cache_size_kb : 131072
 properties.max_device_pinned_mem_size_kb : 33554432
 properties.posix_pool_slab_size_kb : 4 1024 16384
 properties.posix_pool_slab_count : 128 64 32
 properties.rdma_peer_affinity_policy : RoundRobin
 properties.rdma_dynamic_routing : 0
 fs.generic.posix_unaligned_writes : false
 fs.lustre.posix_gds_min_kb: 0
 fs.beegfs.posix_gds_min_kb: 0
 fs.weka.rdma_write_support: false
 fs.gpfs.gds_write_support: false
 profile.nvtx : false
 profile.cufile_stats : 0
 miscellaneous.api_check_aggressive : false
 execution.max_io_threads : 4
 execution.max_io_queue_depth : 128
 execution.parallel_io : true
 execution.min_io_threshold_size_kb : 8192
 execution.max_request_parallelism : 4
 properties.force_odirect_mode : false
 properties.prefer_iouring : false
 =========
 GPU INFO:
 =========
 GPU index 0 NVIDIA A800-SXM4-80GB bar:1 bar size (MiB):131072 supports GDS，IOMMU State: Disabled
 GPU index 1 NVIDIA A800-SXM4-80GB bar:1 bar size (MiB):131072 supports GDS，IOMMU State: Disabled
 GPU index 2 NVIDIA A800-SXM4-80GB bar:1 bar size (MiB):131072 supports GDS，IOMMU State: Disabled
 GPU index 3 NVIDIA A800-SXM4-80GB bar:1 bar size (MiB):131072 supports GDS，IOMMU State: Disabled
 GPU index 4 NVIDIA A800-SXM4-80GB bar:1 bar size (MiB):131072 supports GDS，IOMMU State: Disabled
 GPU index 5 NVIDIA A800-SXM4-80GB bar:1 bar size (MiB):131072 supports GDS，IOMMU State: Disabled
 GPU index 6 NVIDIA A800-SXM4-80GB bar:1 bar size (MiB):131072 supports GDS，IOMMU State: Disabled
 GPU index 7 NVIDIA A800-SXM4-80GB bar:1 bar size (MiB):131072 supports GDS，IOMMU State: Disabled
 ==============
 PLATFORM INFO:
 ==============
 IOMMU: disabled
 Platform verification succeeded

Because there is no additional storage system that supports GDS, only the NVMe device is in a GDS-supported state.

4. Testing GDS

4.1 Creating a 100G File

1

time dd if=/dev/zero of=/data/test/dd.txt bs=40M count=2500

4.2 Testing Under Different Transfer Models

/usr/local/cuda-12.2/gds/tools/gdsio provides 8 different test modes, namely:

  • 0 - Storage->GPU (GDS)

GPU Direct Storage (GDS): data is transferred directly from storage to the GPU, bypassing the CPU. This mode reduces latency and increases throughput, and is especially suited to scenarios that need high-speed GPU access to data.

1
2
3

/usr/local/cuda-12.2/gds/tools/gdsio -f /data/test/dd.txt -d 0 -w 4 -s 10G -i 1M -I 0 -x 0

IoType: READ XferType: GPUD Threads: 4 DataSetSize: 10203136/10240000(KiB) IOSize: 1024(KiB) Throughput: 4.443489 GiB/sec，Avg_Latency: 878.805571 usecs ops: 9964 total_time 2.189826 secs
  • 1 - Storage->CPU

The traditional storage-to-CPU transfer: data is first transferred from storage to the CPU. This is the traditional approach, where the CPU receives and processes the data and then passes it on to other components, such as the GPU.

1
2
3

/usr/local/cuda-12.2/gds/tools/gdsio -f /data/test/dd.txt -d 0 -w 4 -s 10G -i 1M -I 0 -x 1

IoType: READ XferType: CPUONLY Threads: 4 DataSetSize: 10240000/10240000(KiB) IOSize: 1024(KiB) Throughput: 4.414133 GiB/sec，Avg_Latency: 868.366600 usecs ops: 10000 total_time 2.212354 secs
  • 2 - Storage->CPU->GPU

Storage to CPU to GPU: data is first transferred from storage to the CPU, processed, and then transferred to the GPU. This is the more common flow, and it suits scenarios that need the CPU to preprocess data.

1
2
3

/usr/local/cuda-12.2/gds/tools/gdsio -f /data/test/dd.txt -d 0 -w 4 -s 10G -i 1M -I 0 -x 2

IoType: READ XferType: CPU_GPU Threads: 4 DataSetSize: 10142720/10240000(KiB) IOSize: 1024(KiB) Throughput: 4.312804 GiB/sec，Avg_Latency: 905.410136 usecs ops: 9905 total_time 2.242822 secs
  • 3 - Storage->CPU->GPU_ASYNC

Asynchronous storage to CPU to GPU transfer: data is first transferred from storage to the CPU, and then transferred asynchronously to the GPU. This mode lets the CPU perform other tasks while waiting for the data transfer to complete, improving efficiency.

1
2
3

/usr/local/cuda-12.2/gds/tools/gdsio -f /data/test/dd.txt -d 0 -w 4 -s 10G -i 1M -I 0 -x 3

IoType: READ XferType: CPU_ASYNC_GPU Threads: 4 DataSetSize: 10167296/10240000(KiB) IOSize: 1024(KiB) Throughput: 3.318594 GiB/sec，Avg_Latency: 1176.616969 usecs ops: 9929 total_time 2.921806 secs
  • 4 - Storage->PAGE_CACHE->CPU->GPU

Storage->page cache->CPU->GPU: data first enters the operating system’s page cache, then is transferred to the CPU, and finally to the GPU. The page cache can speed up access to frequently used data, but it adds an extra caching step.

1
2
3

/usr/local/cuda-12.2/gds/tools/gdsio -f /data/test/dd.txt -d 0 -w 4 -s 10G -i 1M -I 0 -x 4

IoType: READ XferType: CPU_CACHED_GPU Threads: 4 DataSetSize: 10011648/10240000(KiB) IOSize: 1024(KiB) Throughput: 1.770729 GiB/sec，Avg_Latency: 2206.536591 usecs ops: 9777 total_time 5.392046 secs
  • 5 - Storage->GPU_ASYNC

Asynchronous storage-to-GPU transfer: data is transferred asynchronously from storage directly to the GPU, allowing other operations to continue during the transfer, reducing wait time and improving overall processing efficiency.

1
2
3

/usr/local/cuda-12.2/gds/tools/gdsio -f /data/test/dd.txt -d 0 -w 4 -s 10G -i 1M -I 0 -x 5

IoType: READ XferType: ASYNC Threads: 4 DataSetSize: 10160128/10240000(KiB) IOSize: 1024(KiB) Throughput: 3.053654 GiB/sec，Avg_Latency: 1272.345376 usecs ops: 9922 total_time 3.173068 secs
  • 6 - Storage->GPU_BATCH

Batched storage-to-GPU transfer: data is transferred from storage to the GPU, but in a batched manner. It suits cases where a large amount of data needs to be processed at once.

1
2
3

/usr/local/cuda-12.2/gds/tools/gdsio -f /data/test/dd.txt -d 0 -w 4 -s 10G -i 1M -I 0 -x 6

IoType: READ XferType: GPU_BATCH Threads: 1 IoDepth: 4 DataSetSize: 10240000/10240000(KiB) IOSize: 1024(KiB) Throughput: 4.422783 GiB/sec，Avg_Latency: 873.240876 usecs ops: 10001 total_time 2.208027 secs
  • 7 - Storage->GPU_BATCH_STREAM

Streaming batched storage-to-GPU transfer: this is a mode that combines batching and streaming; as data is transferred from storage to the GPU, streaming is used to batch it, further optimizing processing speed and resource utilization.

1
2
3

/usr/local/cuda-12.2/gds/tools/gdsio -f /data/test/dd.txt -d 0 -w 4 -s 10G -i 1M -I 0 -x 7

IoType: READ XferType: GPU_BATCH_STREAM Threads: 1 IoDepth: 4 DataSetSize: 10240000/10240000(KiB) IOSize: 1024(KiB) Throughput: 2.437489 GiB/sec，Avg_Latency: 1600.979502 usecs ops: 10001 total_time 4.006429 secs

4.3 Test Results Under Different Modes

ModeSpeed (GiB/sec)
0 - Storage->GPU (GDS)4.443489 GiB/sec
1 - Storage->CPU4.414133 GiB/sec
2 - Storage->CPU->GPU4.312804 GiB/sec
3 - Storage->CPU->GPU_ASYNC3.318594 GiB/sec
4 - Storage->PAGE_CACHE->CPU->GPU1.770729 GiB/sec
5 - Storage->GPU_ASYNC3.053654 GiB/sec
6 - Storage->GPU_BATCH4.422783 GiB/sec
7 - Storage->GPU_BATCH_STREAM2.437489 GiB/sec
  • Storage->GPU (GDS)\Storage->GPU_BATCH: the fastest, because data is transferred directly from storage to the GPU, bypassing the CPU
  • Storage->CPU->GPU_ASYNC and Storage->GPU_ASYNC: slightly slower in asynchronous mode, because asynchronous transfer adds management overhead
  • Storage->PAGE_CACHE->CPU->GPU: the slowest, because the data goes through the page cache, adding an extra step

4.3 Verifying the Low CPU Utilization in GDS Mode

Mainly comparing mode 0 and mode 2

  • 0 - Storage->GPU (GDS)
1
2
3

/usr/local/cuda-12.2/gds/tools/gdsio -f /data/test/dd.txt -d 0 -w 40 -s 100G -i 1M -I 0 -x 0

IoType: READ XferType: GPUD Threads: 40 DataSetSize: 101975040/102400000(KiB) IOSize: 1024(KiB) Throughput: 6.653494 GiB/sec，Avg_Latency: 5870.442542 usecs ops: 99585 total_time 14.616528 secs

1
2

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
43261 root      20   0   41.9g 753364 700692 D  18.8   0.1   0:04.69 gdsio
  • 2 - Storage->CPU->GPU
1
2
3

/usr/local/cuda-12.2/gds/tools/gdsio -f /data/test/dd.txt -d 0 -w 40 -s 100G -i 1M -I 0 -x 2

IoType: READ XferType: CPU_GPU Threads: 40 DataSetSize: 101849088/102400000(KiB) IOSize: 1024(KiB) Throughput: 6.649129 GiB/sec，Avg_Latency: 5874.226678 usecs ops: 99462 total_time 14.608058 secs

1
2

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
47977 root      20   0   36.0g 153276 138644 S  63.7   0.0   0:03.27 gdsio

As you can see, in GDS mode CPU utilization is only 30% of that of the transfer through the CPU.

5. References

微信公众号
WRITTEN BY
微信公众号

See Also