1. About FLOPs
FLOPs (Floating Point Operations Per Second) refers to the number of floating-point operations performed per second.
Specifically:
- A single floating-point addition, such as a + b, counts as one floating-point operation.
- A single floating-point multiplication, such as a * b, also counts as one floating-point operation.
- Other basic floating-point operations, such as division and square root, can also be counted as one floating-point operation.
2. Single and Double Precision
Under the format defined by ANSI/IEEE Std. 754-1985, a format used for 32-bit numbers is called single precision, and one used for 64-bit numbers is called double precision.
32-bit (single-precision) format
| Sign bit | Exponent | Mantissa |
|---|---|---|
| 1 bit | 8 bits | 23 bits |
64-bit (double-precision) format
| Sign bit | Exponent | Mantissa |
|---|---|---|
| 1 bit | 11 bits | 52 bits |
3. Common Units
| Unit | Definition |
|---|---|
| MFLOPS | One million ((10^6)) floating-point operations per second |
| GFLOPS | One billion ((10^9)) floating-point operations per second |
| TFLOPS | One trillion ((10^{12})) floating-point operations per second |
| PFLOPS | One quadrillion ((10^{15})) floating-point operations per second |
| EFLOPS | One quintillion ((10^{18})) floating-point operations per second |
4. Common Devices and Compute Power
| Device model | Single-precision compute (FP32) |
|---|---|
| GeForce RTX 4090 | 82.58 TFLOPS |
| NVIDIA H100 | 51.22 TFLOPS |
| NVIDIA H20 | 44 TFLOPS |
| NVIDIA A100 | 19.5 TFLOPS |
| NVIDIA A800 | 19.5 TFLOPS |
| Tesla V100 | 15.7 TFLOPS |
This means a GeForce RTX 4090 can perform 82.58 trillion single-precision floating-point operations per second.
