1. What Is Model Quantization
Model quantization is the process of converting the weights and activations of a high-precision model (usually 32-bit floating point FP32 or 16-bit floating point FP16) into a low-precision model (such as 8-bit integer INT8).
The value range of FP32 is -3.4*10^38 to 3.4*10^38, with 4 billion values. For INT8, on the other hand, we can only see 256 values within the set of possible values, with a range of -128 to 128. Computing INT8 matrices is much faster than FP32 and FP16.
The process of model quantization is establishing a mapping from high-precision values to low-precision values.
2. Benefits of Model Quantization
- Reducing the model’s VRAM footprint
Take https://huggingface.co/Qwen/Qwen2.5-7B as an example. It uses bfloat16 precision by default. Let us estimate that the VRAM occupied by the model weights is 7B _ 16/8 Byte = 14 GB of VRAM. If we perform Int8 quantization, the VRAM occupied by the model weights drops to 7B _ 8/8 Byte = 7 GB of VRAM, only half of what it was before.
- Improving the model’s inference speed
First, the amount of computation is greatly reduced: W8A16 (weights 8-bit, activations 16-bit) reduces computation by 30~40%, and W8A8 (weights 8-bit, activations 8-bit) reduces computation by 60-75%.
Second, the amount of data transfer is reduced by around 50%, which effectively lowers VRAM bandwidth usage. To transfer the same amount of data, the quantized version of the model takes much less time than the non-quantized version.
3. Common Quantization Bit Widths
| Abbreviation | Meaning | Use Cases |
|---|---|---|
| W16A16 | Weights FP16/BF16, activations FP16/BF16 | Common mixed-precision training and inference |
| W8A8 | Weights INT8, activations INT8 | Classic INT8 quantization, widely used in inference deployment (TensorRT, ONNX Runtime, OpenVINO, etc.) |
| W8A16 | Weights INT8, activations FP16/BF16 | A compromise that reduces weight storage and bandwidth while keeping higher precision |
| W4A16 | Weights INT4, activations FP16/BF16 | A common configuration for LLM inference (such as GPTQ, AWQ, QLoRA) |
| W4A8 | Weights INT4, activations INT8 | An aggressive low-precision approach with a higher compression ratio but a greater risk of precision loss |
4. Common Quantization Methods
4.1 GGUF
GGUF stands for GPT-Generated Unified Format, a large model file format defined and released by Georgi Gerganov (founder of the llama.cpp open-source project).
Principles:
- Uses a unified format generated by GPT
- Employs block-level quantization, quantizing weights in blocks
- Balances model size and precision loss through the block structure
Features and advantages:
- Supports multiple quantization precisions (Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, etc.)
- Low hardware requirements; runs on an ordinary CPU
- Relatively small model files, easy to store and transfer
- Strong community support and a mature toolchain
- Supports streaming generation for a good user experience
- Uses block-level quantization, effectively balancing model size and precision
Use cases:
- CPU inference
- Running large models on a local personal computer
- Edge device deployment
- Resource-constrained environments
4.2 GPTQ
GPTQ stands for Gradient-based Post-Training Quantization, a gradient-based post-training quantization method.
Principles:
- Performs post-training quantization based on gradient information
- Uses gradient descent to optimize the error in the quantization process
- Uses gradient information to guide the selection of quantization parameters
Features and advantages:
- Gradient-based post-training quantization technique
- Usually quantizes to 4bit or 8bit
- Still maintains good model performance at 4bit quantization
- Significantly reduces VRAM usage compared with the original model
- Fast inference
- Suitable for deploying large models on consumer-grade GPUs
Use cases:
- GPU inference
- Production environments that require higher precision
- Medium-scale inference services
4.3 AWQ
AWQ stands for Activation-aware Weight Quantization, an activation-aware weight quantization method.
Principles:
- Performs weight quantization based on the activation distribution
- Uses gradient descent to optimize the quantization error
- Determines quantization parameters by minimizing the quantization error
- Quantization parameters include the quantization range, quantization step size, and so on
Features and advantages:
- Activation-aware weight quantization technique
- Protects important weight channels from being quantized, preserving key information
- Usually achieves 4bit quantization
- Higher precision at the same quantization bit width
- Fast inference and low VRAM usage
- Especially suitable for applications that need high-quality output
Use cases:
- GPU inference
- High-performance inference needs
- Scenarios with high requirements on model precision
4.4 FP4
FP4 stands for 4-bit Floating Point, a 4-bit floating-point representation.
Principles:
- Uses a 4-bit floating-point format to represent weights
- Preserves the exponent and mantissa structure of floating point
- Preserves the original value distribution better than integer quantization
Features and advantages:
- 4-bit floating-point representation, preserving numerical precision
- Extremely low storage and computation overhead
- Better precision than 4bit integer quantization
- Preserves the value distribution better than integer quantization
- Suitable for large-scale deployment
- Usually combined with special hardware optimizations
Use cases:
- Extremely resource-constrained environments
- Scenarios that need to maximize the compression ratio
- Specific hardware accelerators
4.5 NVFP4
NVFP4 stands for NVIDIA 4-bit Floating Point, a 4-bit floating-point representation specifically optimized by NVIDIA.
Principles:
- A 4-bit floating-point format optimized for NVIDIA hardware
- Specifically optimized for NVIDIA GPU architectures
- Achieves extreme compression while preserving floating-point characteristics
Features and advantages:
- NVIDIA’s dedicated 4-bit floating-point representation
- Extremely low storage and computation overhead
- Better precision than 4bit integer quantization
- Preserves the value distribution better than integer quantization
- Suitable for large-scale deployment
- Specifically optimized for NVIDIA hardware
Use cases:
- Extremely resource-constrained scenarios in NVIDIA GPU environments
- NVIDIA hardware deployments that need to maximize the compression ratio
- NVIDIA-specific hardware accelerators
