1. Building the nccl-test Image Check the CUDA version 1 2 3 nvidia-smi | grep "CUDA Version" | awk '{print $9}' 12.2 Write the Dockerfile 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cat > Dockerfile << EOF FROM nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04 ENV DEBIAN_FRONTEND=noninteractive ARG CONDA_VERSION WORKDIR /workspace ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt install -y openmpi-bin libopenmpi-dev ssh openssh-server net-tools vim git iputils-ping nfs-common RUN git clone https://github.

Content referenced or recorded while handling faults, continuously updated. 1. XID Error Events XID is NVIDIA’s error code. You can retrieve it with the command: 1 dmesg -T | grep -i "NVRM: Xid" or 1 journalctl --since `date -d "10 days ago" "+%Y-%m-%d"`|grep Xid The XID lets you locate the fault.

1. What Is TensorRT TensorRT is a C++ library mainly used for high-performance inference acceleration on NVIDIA GPUs. It provides both a C++ API and a Python API for integration. The mainstream deep learning frameworks TensorRT supports are: Caffe, where TensorRT can read the prototxt format directly TensorFlow, where the

1. What Is nvidia-smi nvidia-smi stands for NVIDIA System Management Interface, the interface NVIDIA provides for managing and monitoring GPUs. nvidia-smi calls NVML. NVML stands for NVIDIA Management Library, a library that provides a set of C APIs for NVIDIA GPU monitoring and management. 1.1 Queryable State ECC error counts GPU utilization Active compute processes Clocks and PState Temperature and fan speed Power management Hardware identification 1.

If you use nvidia-container-runtime, this is worth your attention — especially if JuiceFS is also in the picture. 1. An Alert Out of Nowhere, and I Panicked Over the weekend I was studying TensorRT LLM, and while I was at it I installed Dragonfly on the largest production cluster. Then