1. Storage Tiering and Interconnect 2. Various Storage Types Storage Type Bandwidth Capacity Response Latency Cost per Unit of Storage Storage Principle Use Case CPU L1 Cache 256-512GB/s 32-64KB/core <1ns ~$2000/MB SRAM Closest cache layer to CPU core CPU L2 Cache 64-256GB/s 256KB-2MB/core group 2-5ns ~$1000/MB Partly SRAM, partly embedded DRAM Next-level cache for the CPU CPU L3 Cache 32-128GB/s 4-16MB/CPU 5-15ns ~$100-200/MB Embedded DRAM Last-level cache for the CPU GPU L1 Cache 8-32TB/s 16-96KB/SM <1ns ~$2000/MB SRAM Closest cache layer to GPU core GPU L2 Cache 1-4TB/s 512KB-4MB/SMC 5-15ns ~$500-1000/MB Partly SRAM, partly embedded DRAM Next-level cache for the GPU Main memory (RAM) 10-80GB/s 4-128GB 30-60ns ~$2-7/GB DDR4/5 DRAM System main storage space GPU VRAM 768GB/s-4TB/s 4-24GB 25-75ns ~$7-20/GB GDDR6/6X SDRAM Dedicated high-speed buffer storage for the GPU SSD 500MB/s-7000MB/s 128GB-8TB 0.