1 Volatile Memory

1.1 REG

REG (Register) is a storage unit inside the CPU, sitting right next to the arithmetic unit — the fastest and smallest level of the storage hierarchy. General-purpose registers (RAX/RBX/RDI on x86, and so on) take part directly in instruction execution, with access latency around 0.2–0.5 ns (roughly one clock cycle), lost on power loss, and typically only tens to hundreds of bytes.

Characteristics:

• Fastest: inside the CPU, wired straight to the arithmetic unit — the quickest data the CPU can reach
• Tiny capacity: usually tens to hundreds of bytes, counted in individual registers
• Lost on power loss; a volatile memory
• Visible to programmers and compilers; register allocation is decided by the compiler and calling conventions

Where it fits:

• Holding intermediate values, operands, and addresses for instruction execution
• Parameter passing and function calling conventions (ABI)
• Local hot data in performance-critical code

Caveats:

• Registers are CPU hardware resources that operations does not manage directly, but understanding them helps with process context switching and performance profiling (perf, say)
• Register state must be saved and restored on a context switch; frequent switching carries overhead
• Limited in number, so not usable as bulk storage

Inspect:

1

grep -m1 flags /proc/cpuinfo

1.2 SRAM

SRAM (Static Random Access Memory) stores bits in flip-flop circuits and does not need the periodic refresh DRAM requires, which makes access faster (around 0.5–2 ns) but also more expensive and more area-hungry. It is typically used for CPU caches (L1/L2/L3) and as the cache layer behind registers. Lost on power loss; a volatile memory.

Characteristics:

• Fast, no refresh: access latency typically an order of magnitude below DRAM (a few ns)
• Expensive, low density: larger circuit cells, so far less capacity for the same area than DRAM
• No refresh needed: bistable circuits hold data, but data is still lost on power loss
• Normally used as cache, not as main memory

Where it fits:

• CPU caches (L1 / L2 / L3)
• High-speed buffers and internal CPU structures such as the TLB
• Scratch data that needs extremely low latency

Caveats:

• SRAM is generally not main memory: small capacity, high cost
• What operations cares about is the cache level it lives in: cache hit rate affects performance, and multi-level cache (L1/L2/L3) grows larger but slower at each step
• Checking cache size and hierarchy can guide performance tuning

Inspect:

1
2

lscpu | grep -iE 'cache|model name'
cat /sys/devices/system/cpu/cpu0/cache/index*/size

1.3 DRAM

DRAM (Dynamic Random Access Memory) is the CPU’s scratch space, lost on power loss. It is byte-addressable and randomly accessible, with capacity typically in the tens to hundreds of GB; bandwidth depends on the DDR generation and the number of memory channels, and latency is around 60–100 ns. Compared with disks, memory is the “fast device,” and it is where Page Cache, tmpfs, io_uring queues and the like live.

Characteristics:

• Byte-addressable, readable and writable in place, with none of NAND’s erase-before-write
• Latency in the ns range, orders of magnitude faster than NVMe (μs range); capacity far smaller than VRAM or disks
• Bandwidth limited by DDR generation and channel count; only multiple parallel channels yield high bandwidth
• Data is lost on power loss, so it is not persistent storage; ECC memory can correct single-bit errors

Where it fits:

• Application working memory, JVM / database buffer pools
• Page Cache (the caching layer for file reads and writes)
• tmpfs / /dev/shm (temporary directories)

Caveats:

• OOM: when memory runs short the kernel’s OOM Killer may kill critical processes; check /proc/meminfo and cgroup limits first
• With swap enabled, memory pressure lands on disk, so “memory being slow” is really the disk being slow
• NUMA: on multi-socket servers, memory is spread across nodes, and cross-node access loses bandwidth; CPU and NUMA pinning avoids this
• ECC and memory faults: watch edac / mcelog — memory errors accumulate and gradually destabilize the system

Inspect:

1
2
3
4

free -h
cat /proc/meminfo
numactl --hardware
dmidecode -t memory | grep -E 'Size|Speed|Type:'

1.4 VRAM

VRAM (Video Random Access Memory) is the GPU’s own high-speed memory, holding model parameters, intermediate activations, KV cache and so on, with access latency around 100–200 ns. Data-center GPUs mostly use HBM for VRAM, consumer GPUs use GDDR. VRAM is physically separate from system memory, and data must be copied between the two over PCIe.

Characteristics:

• Capacity: A100 40/80 GB, H100 80 GB, H200 141 GB; consumer cards like the RTX 4090 have 24 GB
• Extremely high bandwidth: about 3.35 TB/s on H100, several times system memory bandwidth
• The GPU accesses VRAM directly, without going through CPU memory
• When VRAM runs short there is no graceful degradation as with swap — usually a straight CUDA OOM

Where it fits:

• Large-model training: parameters, gradients, optimizer state, activations
• Inference: model weights plus KV cache (vLLM / TensorRT-LLM)
• Storage-adjacent: GDS reads checkpoints from NVMe straight into VRAM, skipping the memory hop

Caveats:

• Common CUDA OOM triggers: KV cache too large, too many concurrent processes, fragmentation; check nvidia-smi first
• Multiple processes or containers sharing VRAM need isolation and MPS / MIG
• When VRAM is short you can use CPU offload or tensor parallelism, at the cost of throughput

Inspect:

1
2

nvidia-smi
nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv

1.5 HBM

HBM (High Bandwidth Memory) is a 3D-stacked DRAM, vertically interconnected through TSVs and accessed over a wide bus, designed specifically for high bandwidth, with access latency around 100–200 ns. The VRAM of data-center GPUs (A100 / H100 / H200 / MI300) is basically all HBM.

Characteristics:

• High bandwidth: a single HBM3e stack can reach 1.2 TB/s+, and an H200 card about 4.8 TB/s
• Relatively small capacity, high cost: limited by stack count and yield, capacity is often below GDDR
• Difference from GDDR: GDDR uses a separate memory bus, with large capacity but comparatively low bandwidth; HBM uses a wide bus, for very high bandwidth and lower power
• Mounted on the same substrate as the GPU, so it does not consume PCIe bandwidth

Where it fits:

• High-bandwidth needs in large-model training and inference, especially where VRAM bandwidth is the limit
• Cases where high bandwidth and low power beat GDDR (standard on data-center GPUs)

Caveats:

• HBM faults are usually whole-card problems, observed via nvidia-smi / ECC counters
• High bandwidth does not replace capacity; when bandwidth saturates it becomes the training or inference throughput bottleneck
• Decide whether the bottleneck is VRAM capacity or VRAM bandwidth by looking at Used versus utilization in nvidia-smi

Inspect:

1
2

nvidia-smi -q -d MEMORY
nvidia-smi -q | grep -iE 'HBM|Memory|Bandwidth'

2 Non-Volatile Memory

2.1 HDD

An HDD (Hard Disk Drive) records information using magnetic field direction. Platters are coated with magnetic material, the head floats just above the surface, and the field direction represents 0/1. Platters are divided into concentric tracks, then sliced into sectors (commonly 4K). To read or write, the head first seeks laterally to the target track, then waits for the target sector to rotate under it.

It can rewrite in place, unlike NAND which must erase before writing. Latency comes mainly from seek and rotational wait, around 5–15 ms; at 7200rpm half a rotation takes about 4 ms. Sequential reads and writes are therefore acceptable, while random access is poor.

Characteristics:

• Large capacity, low unit price, commonly 4–28 TB
• Acceptable sequential throughput, 100–280 MB/s
• Poor random throughput, 50–200 IOPS
• Latency 5–15 ms
• Utilization can reach 80–90%

Where it fits:

• Data backup
• Large-capacity / large-file storage
• Cold data, archives, sequential logs

Inspect devices:

1
2

lsblk -d -o NAME,ROTA,TRAN,MODEL,SIZE
smartctl -a /dev/sdX

ROTA=1 usually means spinning rust. /dev/sdX is not exclusive to HDDs — SATA SSDs, SAS, and virtio also show up under that name.

Caveats:

• Consumer drives are mostly SATA; data centers commonly use SAS (12Gb/s, expanders, multipath)
• Watch SMART reallocated / pending counts, vibration, and bad sectors
• Use the mq-deadline / bfq schedulers; RAID rebuilds, fsck, and full-disk scans will saturate the drive
• Do not put WAL or system disks on HDDs, and do not bind fast and slow disks into one RAID / LVM

2.2 SSD

On an SSD (Solid State Drive) data lands on NAND flash: written by page, erased by block, with no in-place rewrite. The FTL writes new data to free pages, invalidates the old ones, and waits for GC to erase whole blocks.

SLC cache absorbs writes at high speed first, then flushes them back to TLC / QLC; once the cache fills or the drive gets full, speed drops to the native level. Access latency is around 50–200 μs, and GC, wear leveling, and cache flush all cause latency spikes — the rated speed only holds while the cache has room.

Characteristics:

• SATA SSDs commonly 480 G–3.84 T; sequential about 500–560 MB/s (capped by the interface); random 4K about 50–100K IOPS; latency 50–200 μs
• QLC commonly 4–8 T+; burst can approach TLC, sustained writes often fall to hundreds of MB/s, with high write-latency jitter
• Keep utilization below 80%, and leave QLC more headroom still

Where it fits:

• Read-heavy, write-light capacity workloads such as objects and images (QLC)
• System disks and light databases (SATA SSD, especially with no NVMe slot)
• Continuous rewrite and database logs are not suited to cheap QLC

Caveats:

• Judge lifetime by percentage_used / TBW, write amplification, and thermal throttling
• Enterprise drives have PLP and can flush back to NAND on power loss; consumer drives may lose data
• Specs like 3.84T exist to leave OP for GC
• Use scheduled fstrim, not a continuous discard mount option

1
2

smartctl -a /dev/sdX
fstrim -v /

2.3 NVMe

NVMe (Non-Volatile Memory Express) is a storage protocol designed for flash, supporting multiple queues and deep queues, with access latency around 10–100 μs. Locally it usually runs over PCIe; across machines it becomes NVMe-oF (RDMA or TCP). Devices are typically /dev/nvme0n1.

Compared with SATA / AHCI: the NVMe spec allows up to 64K queues, far more than real drives use. SATA SSD performance is limited by NCQ=32 and about 600 MB/s.

Characteristics:

• TLC capacity commonly 1.92–7.68 T (15 T+ exists); sequential 2–14 GB/s (Gen3 to Gen5); random 4K about 200K–1.5M IOPS; latency 10–100 μs
• Keep utilization below 70–80%
• QLC NVMe reads acceptably, but writes are weak and unstable
• NVMe-oF (RDMA) throughput is often limited by 25/100/200GbE, adding about 20–100 μs over local, still slower than local NVMe

Where it fits:

• Capacity workloads, read-heavy (QLC NVMe)
• Shared disks and compute-storage separation (NVMe-oF)
• Databases, WAL, virtual machines, high IOPS (TLC NVMe)

Inspect devices:

1
2
3
4

lsblk -o NAME,TRAN,TYPE
nvme list
nvme smart-log /dev/nvme0
cat /sys/block/nvme0n1/queue/scheduler

Caveats:

• Use the none or mq-deadline scheduler
• Queue depth, interrupt pinning, and blk-mq all affect saturated IOPS

3 Data Transfer

3.1 SATA

SATA (Serial ATA) is a local serial interface paired with AHCI. It is point-to-point, one port per drive. Shared bandwidth is more common with SAS expanders, or when several ports on a chipset contend for total bandwidth.

Characteristics:

• SATA III is theoretically 6Gb/s, practically about 550–600 MB/s
• NCQ queue depth 32
• Devices are usually /dev/sdX, sharing names with SCSI and virtio

Where it fits:

• HDDs, early or entry-level SSDs
• Sequential access is adequate; random and high-concurrency are weaker than NVMe

Caveats:

• Do not let AHCI get set to IDE compatibility mode in the BIOS
• Motherboard ports, HBAs, and cables can all be bottlenecks
• Rough ordering: HDD < SATA SSD < RDMA network disk < local NVMe

Inspect devices:

1

lsblk -d -o NAME,TRAN,SIZE

TRAN=sata means it is actually on SATA. The device name may still be /dev/sdX.

3.2 PCIe

PCIe (Peripheral Component Interconnect Express) is a point-to-point serial bus whose bandwidth is decided jointly by the number of lanes (x1 / x2 / x4 / x8 / x16) and the generation. NVMe drives are usually ×4, while cheap M.2 slots are often ×2. One way to think of it: PCIe is the transport channel, and NVMe is the protocol running on top of it.

Approximate unidirectional bandwidth (GB/s), already accounting roughly for encoding:

Data centers commonly run Gen3 / Gen4 today and Gen5 on new machines; Gen2 persists on old platforms and Gen6 is just starting. For NVMe, read the ×4 column; GPUs and high-speed NICs commonly use ×8 or ×16. It is full-duplex, and the real ceiling is further limited by controllers, NAND, and cooling.

Where it fits:

• Local NVMe, GPUs, high-speed NICs
• Anywhere GB/s-scale local bandwidth is needed and SATA is not enough

Inspect the link:

1

lspci -vv | grep -A2 -E 'LnkCap|LnkSta'

Compare Speed / Width between LnkSta (actual) and LnkCap (capability). A wrong slot, lane contention, or a riser can cause a generation downgrade or a halved width — Gen4 ×4 becoming Gen3 ×2, for instance. Gen5 drives tend to hit thermal limits first.

3.3 DMA

DMA (Direct Memory Access) lets devices move data directly into and out of memory, with the CPU only submitting descriptors rather than doing the moving. Local NVMe, NICs, and GPU reads and writes all go through DMA; GPUDirect (GDR / GDS) is P2P DMA over PCIe, letting devices transfer directly to each other without a detour through CPU memory.

Characteristics:

• Local disk and NIC access to host memory is DMA underneath
• GPUDirect RDMA (GDR): NIC ↔ GPU VRAM, so RoCE traffic need not enter DRAM
• GPUDirect Storage (GDS): NVMe ↔ GPU VRAM, removing one cudaMemcpy when reading from disk
• Sensitive to PCIe topology; if the GPU and NIC are not under the same switch, P2P may fall back to routing through CPU memory
• Relation to 3.4 RDMA: RDMA is remote DMA; GDR swaps RDMA’s peer from host memory to GPU VRAM

Where it fits:

• Data movement for all high-speed block devices and NICs
• Multi-machine GPU training, with parameters and gradients over RoCE and GPUDirect RDMA enabled
• Large-model checkpoints read straight from NVMe to GPU (GDS)

Caveats:

• GDR requires the NVIDIA driver, nvidia-peermem, and a NIC that supports peer memory
• When GDR fails it often silently falls back to host memory — performance drops noticeably without necessarily reporting an error

Inspect the topology:

1
2

nvidia-smi topo -m
lsmod | grep nvidia_peermem

To see whether P2P is available, first check in nvidia-smi topo -m whether the GPU and NIC sit under the same switch.

3.4 RDMA

RDMA (Remote Direct Memory Access) lets one machine read and write another machine’s memory directly, skipping the copies of the kernel network stack. The CPU still submits descriptors and handles completion queues. Carriers: InfiniBand, RoCE, and occasionally iWARP.

Characteristics:

• Low latency, low CPU usage, high throughput
• Sensitive to packet loss and to PFC / ECN; packet loss hurts RoCE more visibly than TCP
• NVMe-oF can also run over TCP, in which case it is not RDMA and has higher latency
• Local NVMe is usually still faster, but RDMA beats iSCSI / NFS by a wide margin

Where it fits:

• NVMe-oF, some SANs, distributed storage backends
• Compute-storage separation, shared fast storage
• Unsuitable for lossy Ethernet that has not had PFC / ECN tuned

Caveats:

• Watch Pause frames, PFC deadlock, ECN, CNP, and NIC drivers
• Compare throughput against the NIC’s 25/100/200GbE, not against a local NVMe’s rated numbers

Inspect the link:

1
2

ibdev2netdev
ethtool -S eth0 | grep -iE 'pause|pfc|discard|drop'

When troubleshooting RoCE, look at Pause / PFC first rather than only Ethernet packet loss.

3.5 Switch

The switch is the meeting point of the storage network: NVMe-oF, iSCSI, and RoCE traffic all pass through it, and link quality directly determines storage latency and throughput. From the storage side, the concern is mainly ports and forwarding capacity, not route computation.

Characteristics:

• L2 / L3: storage networks are mostly L2 (VLAN isolated), with L3 routing only across subnets
• Wire-speed forwarding: ideally every port runs at line rate; with insufficient backplane bandwidth, multiple ports share a ceiling
• Port buffers: bursts are absorbed by port buffer, and insufficient buffer means packet loss — fatal for RoCE
• PFC / ECN: RoCE depends on a lossless network, with hop-by-hop backpressure from PFC and congestion marking from ECN; misconfiguration causes deadlock
• IB vs Ethernet: InfiniBand switches do native RDMA with end-to-end flow control; Ethernet switches emulate it via RoCE and need extra PFC / ECN tuning

Where it fits:

• Ethernet switches carrying RoCE (compute-storage separation, NVMe-oF)
• IB switches for high-performance, low-latency RDMA storage and training networks
• Data centers using leaf-spine to scale port count

Caveats:

• A lossless network requires PFC / ECN enabled globally and consistent queue mapping, or PFC deadlock follows
• Mismatched optics, fiber, or port speeds cause downgrades or packet loss
• Before a port saturates, check for broadcast storms, loops, and aggregation misconfiguration
• The switch itself is a single point; provide redundancy through stacking or MLAG

Inspect the link:

1
2

ethtool eth0
ethtool -S eth0 | grep -iE 'pause|pfc|drop|discard'

After logging into the switch, run show interface counters.

4 System IO

An application’s reads and writes do not reach the device directly; they traverse a complete path. By default, data goes through VFS and the filesystem into Page Cache (the kernel cache), then through the block layer and driver, and on to the device over DMA:

application → VFS → filesystem → Page Cache (direct IO bypasses) → block layer → driver → DMA / bus → device

Direct IO only skips the Page Cache layer; the rest of the path is unchanged and still goes through the filesystem and block layer. Modern NVMe uses blk-mq multi-queue scheduling; io_uring improves throughput under high concurrency by reducing system calls.

Inspect the situation:

1
2
3

iostat -x 1
df -h
df -i

4.1 Buffered IO

Buffered IO is the default read and write path for applications. On a read, Page Cache is checked first and a hit returns straight from memory without touching the disk; on a write, data goes into Page Cache first and the kernel’s flusher threads write it back asynchronously based on dirty-page ratio or a time threshold, rather than every write landing on disk immediately. Reads and writes therefore both pass through the kernel cache.

The job of fsync / fdatasync is to wait for data to reach the disk (or at least the drive’s internal cache) — it does not mean “only fsync writes to disk,” since the flusher has been writing in the background all along. Note also that if the disk has no PLP (power-loss protection), a successful fsync may only mean the data reached the drive’s DRAM, which is still vulnerable to power loss.

Characteristics:

• A read hit is extremely fast — what you measured may be memory
• Fast writes are often just writes to memory; once dirty pages pile up, the flush stalls every writer
• dirty_ratio / dirty_bytes, dirty_background_ratio, and dirty_expire_centisecs control how much dirt accumulates and when it is written back

Where it fits:

• General file access, the default path for most applications
• Read-heavy workloads that can use Page Cache
• Cases needing crash consistency, where you still call fsync yourself

Caveats:

• dd / hdparm without iflag=direct measures memory
• Too many dirty pages stalls writers; watch Dirty / Writeback in /proc/meminfo
• Align partitions to 4K, or SSD write amplification worsens
• Do not mount with a continuous discard by default

1
2

grep -E 'Dirty|Writeback|Cached' /proc/meminfo
dd if=/dev/zero of=/tmp/test bs=1G count=1 oflag=direct

4.2 Direct IO

Direct IO bypasses Page Cache through the O_DIRECT flag, transferring data directly between the application’s user buffer and the device without the kernel cache. It only skips the cache layer; the path still goes through VFS → filesystem → block layer → driver. Because the cache is bypassed, reads and writes must be aligned to the logical block size (commonly 4K), and unaligned ones usually return EINVAL.

Characteristics:

• Latency closer to the device, less misled by the illusion of cache hits
• The kernel does not merge or read ahead, so the application must do its own caching
• Metadata and journals may still be buffered, so fsync remains necessary

Where it fits:

• MySQL InnoDB (commonly O_DIRECT)
• Virtual machine disks
• Disk benchmarks with fio --direct=1
• PostgreSQL does not do this by default; it remains buffered IO

Benchmarking:

1

fio --name=randwrite --filename=/dev/nvme0n1 --direct=1 --rw=randwrite --bs=4k --iodepth=32 --runtime=60

Use direct for disk benchmarks; to measure application experience, follow the application’s real pattern and do not mix the two in one comparison. Small or unaligned IO may fail outright.

5 Logical Storage

Logical storage is the layer that combines and divides underlying physical devices (HDD / SSD) into logical volumes, sitting below the filesystem. The canonical representatives are RAID (grouping disks into an array) and LVM (volume management across disks).

5.1 RAID

RAID (Redundant Array of Independent Disks) combines multiple disks into one logical volume in exchange for capacity, performance, or redundancy. It can be done in hardware (HBA / RAID card) or in software (kernel md, Linux mdadm, soft RAID). In disk-management terms it is “build the RAID first, then partition and create the filesystem,” and it exposes itself as a block device just like a single disk.

Common levels:

Characteristics:

• Capacity and redundancy trade off: more redundancy means less usable capacity
• The latency floor is set by a single disk: the array is generally no faster than one member, though random IO can be spread across stripes, and small writes that need cross-disk read-modify-write have higher latency
• Parity write amplification: every small write on RAID 5/6 needs a read-modify-write, hurting write performance and jitter
• Rebuilds saturate IO across the whole group, degrading performance and raising failure risk
• A hardware RAID card is itself a single point; a failed battery or lost power protection means data loss
• Software RAID is simple and needs no extra hardware but consumes CPU; to the layers above, both look like “one disk”

Where it fits:

• RAID 1: system disks and database logs, where redundancy matters and writes are light
• RAID 5/6: capacity-oriented large files and cold data, where rebuild time is acceptable
• RAID 10: database data files, high IOPS with redundancy
• RAID 0 or no RAID when you want pure capacity or speed and do not care about redundancy

Caveats:

• Avoid disks from the same batch in one RAID; failures may cluster
• Track rebuild progress and md array state (degraded / rebuilding)
• Before building a RAID, be clear that RAID is not backup — accidental deletion, viruses, and ransomware still cannot be undone
• With a RAID card, reading individual disk SMART data often requires passthrough or single-disk mode, since the card hides it

Inspect the array:

1
2
3

cat /proc/mdstat
mdadm --detail /dev/md0
smartctl -a -d sat /dev/sdX

5.2 LVM

LVM (Logical Volume Manager) aggregates multiple disks or partitions into a volume group and carves out arbitrarily sized logical volumes from it, for flexible expansion, shrinking, and migration. The hierarchy is: physical volume (PV) → volume group (VG) → logical volume (LV), with a filesystem on top of the LV.

Levels:

• PV (Physical Volume): a whole disk or a partition, initialized with pvcreate
• VG (Volume Group): a collection of PVs, created with vgcreate
• LV (Logical Volume): a volume carved out of a VG, created with lvcreate, formatted and mounted for use

Characteristics:

• Flexible growth and shrink: lvextend / lvreduce adjust the LV online, with the filesystem following via resize2fs / xfs_growfs
• Aggregation across disks: multiple PVs form one large VG, divided as needed, for high capacity utilization
• Supports snapshots, striping, and mirroring among other advanced features
• Provides no redundancy itself: that usually comes from RAID below or replication above; losing a disk can degrade the VG or prevent it from mounting

Inspect volumes:

1
2
3

pvdisplay
vgdisplay
lvdisplay

6 Local Filesystems

A local filesystem is a format built on a local block device (or logical volume), mountable for read and write by only one machine at a time. In findmnt / df -T, entries whose source is /dev/... and whose type is ext4 / xfs / btrfs / zfs are local filesystems.

1
2
3

findmnt -D
df -hT
lsblk -f

6.1 ext4

The most common local filesystem on Linux, built on local block devices. Ubuntu system disks mostly use it by default. The toolchain is mature (e2fsprogs) and it can shrink (though that is fiddly). Suited to system disks, root partitions, and small to medium data disks. For very large disks and highly parallel large files it trails XFS. Its latency is that of the underlying disk, and it cannot be written by multiple machines at once (unless exported through NFS).

1
2

tune2fs -l /dev/nvme0n1p1 | grep -E 'Filesystem volume name|Block size|Inode'
resize2fs /dev/nvme0n1p1

Caveats: mkfs.ext4 wipes the partition; use fstrim on SSDs; check df -i for small files.

6.2 XFS

Also a local filesystem, more common on RHEL and on data-center data disks. Allocation parallelism is better, suiting large files, large capacity, and multi-threaded writes. It cannot shrink in place, and repair requires xfs_repair — do not use fsck.ext4 on it.

It sits at the same level as ext4; choosing between them only affects how one machine manages that disk.

1
2

xfs_info /data
xfs_growfs /data

Caveats: confirm the device before creating the FS; align partitions to 4K.

6.3 Btrfs

Btrfs (B-tree File System) is a CoW (copy-on-write) filesystem on Linux offering subvolumes, snapshots, compression, and checksums, and it is the default for system disks on Fedora-family systems. Data checksumming carries some overhead and write amplification is higher than ext4; snapshots and rollback are friendly to system disks and container images.

1
2
3

btrfs filesystem usage /data
btrfs subvolume list /data
btrfs filesystem df /data

Caveats: check it with btrfs, not fsck.ext4; a snapshot is not a backup — if the original data is corrupted the snapshot may be too.

6.4 ZFS

ZFS (Zettabyte File System) is a filesystem with an integrated storage pool, combining checksums, snapshots, compression, RAID-Z, and caching (ARC / L2ARC), commonly used via OpenZFS and frequently seen on NAS and storage servers. Single-disk devices are usually /dev/zvol or a mount point rather than a raw block device.

1
2
3

zpool status
zpool list
zfs list

Caveats: allocate plenty of memory (ARC consumes it); RAID-Z expansion is complex and usually means rebuilding entirely; L2ARC does not persist across power loss.

7 Network Filesystems

Network filesystems let multiple machines access the same data through a shared interface, while the data still lands on the local filesystem or object storage of some machine or cluster. In findmnt / df -T, types nfs4, ceph, and fuse.juicefs are network filesystems.

7.1 NFS

Exports a directory from one machine, which clients read and write at a local path, with data crossing the network (usually TCP 2049). The server-side directory is still ext4 or XFS underneath. Scaling means giving that machine stronger hardware; if the server fails, clients block along with it.

NFSv3 is stateless and simple; NFSv4.1 has compound operations, delegations, and nconnect. Use 4.1 for new deployments.

Characteristics: one directory across many machines, simple to deploy; latency and throughput are limited by the network and the server’s disks. sync is safer and slower; async may lose recent writes on power loss.

Where it fits: shared configuration, datasets, home directories, backups. Unsuitable for database WAL or high-concurrency random small-file writes.

1
2
3

showmount -e 10.0.0.1
mount -t nfs -o nfsvers=4.1,hard,nconnect=4 10.0.0.1:/data /mnt/nfs
nfsstat -c

Caveats: use hard in production and ensure the server is highly available; run exportfs -ra after editing /etc/exports; no_root_squash effectively maps the client’s root to the server’s root; when things hang, check the server’s disks and network first; a full PVC means checking df on the server.

7.2 CephFS

Ceph’s file interface. Data is stored across multiple OSDs (replicated or erasure-coded), and metadata is managed by MDS. The same cluster can also serve RBD (block) and RGW (S3), but those two are not filesystems.

Difference from NFS: it does not export one machine’s local directory, and capacity grows with OSDs. Difference from JuiceFS: it manages its own disks and does not depend on an external object bucket.

Where it fits: unified block + file + object, sharing within a cluster, and teams able to maintain OSDs. Unsuitable as a single-machine system disk.

1
2
3

df -hT | grep ceph
ceph fs status
ceph -s

Caveats: the bottleneck may be MDS, OSD, or the network; when space runs out, check storage pools / OSDs rather than the df of some NFS machine.

7.3 JuiceFS

Puts POSIX on top of object storage: data lives in S3 / OSS / MinIO, metadata in Redis / TiKV / PostgreSQL and the like, and the client usually has a local cache. The type is generally fuse.juicefs.

Where it fits: training / inference datasets, using a file interface over object storage, read-heavy workloads. Unsuitable for databases.

1
2
3

df -hT | grep juicefs
juicefs status <META-URL>
juicefs stats /mnt/jfs

Caveats: troubleshooting means checking all three layers — local cache, metadata engine, object storage; if the metadata engine fails, directories cannot be listed even though the objects remain in the bucket.

7.4 3FS

3FS (Fire-Flyer File System) is DeepSeek’s open-source distributed filesystem: data is sharded across SSDs on multiple storage nodes, metadata is managed by the Meta Service, and clients pull data over RDMA (RoCE / InfiniBand), targeting the high-throughput reads of large-model training and inference. Like CephFS it is a self-built storage cluster, with capacity scaling by storage node.

Characteristics: high-bandwidth reads aimed at AI, over RDMA, with data and metadata separated; unlike JuiceFS it does not depend on an external object bucket.

Where it fits: multi-GPU nodes sharing training / inference datasets, high-speed checkpoint reads. The deployment bar is higher than NFS, needing an RDMA network and multi-node planning, with limited payoff at small scale.

1
2

df -hT | grep -i 3fs
ethtool -S eth0 | grep -iE 'pause|pfc|discard|drop'

Caveats: the bottleneck is often the RDMA network and the storage nodes’ SSDs; the data plane runs on a lossless network, so check PFC / packet loss first; troubleshooting means checking the client, Meta Service, and storage nodes separately.

8 Operations Monitoring

8.1 snmp_exporter

Collects SNMP from switches and other network devices. Storage does not depend on it directly, but the latency and throughput of storage networks such as NVMe-oF, iSCSI, and RoCE are all affected by switch state, and failures often show up on the link first.

Uses:

• Port status, packet loss, errors, CRC errors
• Port throughput, utilization, optical module TX/RX power
• Power supplies, fans, temperature, device up/down

Metrics:

• Port ifHCInOctets / ifHCOutOctets, ifInErrors / ifOutErrors
• Port link state (up / down / admin down)
• Optical module TX / RX power and temperature

Bottlenecks:

• Sustained port packet loss and CRC errors mean poor link quality
• Abnormal or low optical module power usually means fiber degradation or an impending failure
• Saturated port utilization means investigating the traffic or the link aggregation

1

snmpwalk -v2c -c public 10.0.0.2 .1.3.6.1.2.1.2.2

8.2 ipmi_exporter

Collects hardware monitoring from the motherboard’s IPMI / BMC. Storage nodes have many disks and high power draw, so chassis fan, power supply, and disk backplane temperatures are indirect risk factors for disk lifetime.

Uses:

• Chassis temperature, fan speed, power supply state
• Disk backplane / controller temperature
• Hardware alerts (SEL events, SDR thresholds)

Metrics:

• ipmi_temperature_celsius, ipmi_fan_speed_rpm
• ipmi_power_supply_state
• ipmi_sel_entries

Bottlenecks:

• Fan failure or over-temperature — throttling comes before a crash
• Lost power supply redundancy, lowering the redundancy level
• Piling-up SEL events mean unhandled hardware faults

1
2

ipmitool sdr list
ipmitool sel elist

8.3 smartctl_exporter

Collects SMART from HDD / SSD / NVMe. smartctl now reads NVMe SMART as well, so a separate NVMe text-collection setup is usually unnecessary.

Uses:

• Bad sectors, lifetime, temperature, power loss
• Enterprise drive spare, percentage_used
• Paired with node_exporter: one reflects “performance is slow right now,” the other “is this disk about to fail”

Metrics:

• temperature
• available_spare, percentage_used
• media_errors
• reallocated / pending

Bottlenecks:

• Pending sectors rising means prepare to replace the disk
• Spare dropping to the threshold triggers critical_warning
• Temperature causing throttling
• percentage_used approaching 100% means write amplification is already high

8.4 node_exporter

Covers local capacity, disk IO, memory dirty pages, PSI, and NICs. This is the foundation of storage observability.

Uses:

• Whether filesystems and inodes are full
• Disk throughput, IOPS, busyness
• Whether dirty pages are accumulating, whether tasks are waiting on IO
• NIC / InfiniBand counters (check these too for RDMA and NVMe-oF)

Metrics:

• node_filesystem_avail_bytes, node_filesystem_files_free
• node_disk_read_bytes_total, node_disk_written_bytes_total, node_disk_io_time_seconds_total, node_disk_io_now
• node_memory_Dirty_bytes, node_memory_Writeback_bytes
• node_pressure_io_waiting_seconds_total
• node_network_*, with --collector.infiniband for InfiniBand

Bottlenecks:

• Filesystem or inodes full; the fuller an SSD gets, the slower it is
• node_disk_io_time climbing steadily; write-latency spikes after dirty pages pile up
• High IO PSI means tasks are waiting on disk — not necessarily a CPU shortage
• On multi-queue SSDs, util easily approaches 100% without being saturated. Alerts should watch write latency, queue depth, and the application’s p99
• await must be computed yourself from node_disk_read_time / write_time divided by operation count; node_exporter does not expose iostat’s await directly

8.5 process-exporter

Shows which process is consuming the disk. node_exporter can only say a device is busy, not point at a specific process.

Uses:

• Backups, compaction, log rotation, or one Pod inside a container saturating the disk
• Complementary to cAdvisor / kubelet: processes on the host, PVCs and containers on Kubernetes

Metrics:

• Process read_bytes / write_bytes
• Process IO latency

Bottlenecks:

• A single process saturating writes
• One task flooding dirty pages and stalling other writers
• Watching only node_exporter misses a full PVC; watching only kubelet misses an impending disk failure or SLC cache slowdown. Both the host and the container layer need coverage

9 Performance Metrics

9.1 IOPS

IOPS (Input/Output Operations Per Second) measures how many read and write operations storage completes per second, suited to evaluating random access and small-block IO. Higher is better, but the block size and read/write mix must be stated for the number to mean anything.

Characteristics:

• About operation count, strongly tied to block size and IO pattern (sequential / random)
• Random 4K is the typical case for measuring random IOPS; large sequential blocks are better measured as throughput
• Typical numbers: HDD random around 50–200 IOPS, SATA SSD around 50–100K, NVMe around 200K–1.5M

Inspect:

1

fio --name=randread --filename=/dev/nvme0n1 --direct=1 --rw=randread --bs=4k --iodepth=32 --runtime=60

9.2 Throughput

Throughput measures how much data storage transfers per unit time, in MB/s or GB/s, suited to evaluating sequential access and large-block transfers. It equals “IOPS × block size,” so large sequential IO is mostly a matter of throughput.

Characteristics:

• About data volume, suited to capacity workloads with sequential access
• Affected by interface, protocol, and queue depth: SATA SSD around 500–560 MB/s, NVMe up to 2–14 GB/s
• Complementary to IOPS: on the same link, larger blocks lean toward throughput, smaller blocks toward IOPS

Inspect:

1

fio --name=seqread --filename=/dev/nvme0n1 --direct=1 --rw=read --bs=1M --iodepth=16 --runtime=60

9.3 Latency

Latency measures the time from issuing a single IO to its completion, in ns / μs / ms, reflecting how responsive the storage is. Lower is better, and tail latency (p99 / p99.9) reflects real experience better than the average, since spikes drag down the long tail.

Characteristics:

• Average latency hides jitter; watch p99 / p99.9 tail latency
• Storage types differ by orders of magnitude: memory in ns, NVMe in μs, HDD in ms
• Write amplification, GC, cache flush, and a full SLC cache all cause latency spikes

Inspect:

1

fio --name=lat --filename=/data/fio.test --direct=1 --rw=randread --bs=4k --iodepth=1 --runtime=60

9.4 Bandwidth

Bandwidth measures the theoretical transfer ceiling of a link, related to throughput but focused on the link’s or protocol’s capability. Throughput is what you actually measured, bandwidth is the ceiling; the two being close means the link is nearly saturated.

Characteristics:

• SATA III around 6Gb/s (practically about 550–600 MB/s), PCIe Gen3 ×4 around 4GB/s, Gen5 ×4 around 16GB/s
• Multiple disks or links can be aggregated in parallel, adding bandwidth
• Network storage is further limited by NIC bandwidth (25/100/200GbE, say)

Inspect:

1
2

lspci -vv | grep -A2 -E 'LnkCap|LnkSta'
ethtool eth0 | grep -i speed

9.5 QD

Queue Depth is the number of IOs in flight (not yet complete) at one moment, and it determines whether storage can be saturated. Deep queues favor sequential throughput and high IOPS but add latency; shallow queues suit low-latency scenarios.

Characteristics:

• Queue depth = concurrent in-flight requests, the --iodepth parameter in fio
• Deep queues (32/128, say) extract throughput and IOPS; a shallow queue (1) measures single-request latency
• A trade-off with latency: the deeper the queue, the more likely requests are queued, and latency rises

Inspect:

1

fio --name=qd --filename=/dev/nvme0n1 --direct=1 --rw=randread --bs=4k --iodepth=128 --runtime=60