Red Hat SummitChapter 4. Supported AI accelerator model quantization formats
Different AI accelerator architectures support different types of model quantization, depending on the compute capabilities of the hardware. The following tables list the AI accelerators that support INT8, INT4, FP8, and NVFP4 quantization formats.
- INT8 (W8A8) quantization reduces model weights and activations to 8-bit integers, providing significant memory savings while maintaining acceptable accuracy for many use cases.
- INT8 (W4A8) quantization reduces model weights to 4-bit integers, keeping activations at 8-bit precision. INT8 (W4A8) improves memory efficiency compared to W8A8 while preserving higher activation fidelity for inference.
- INT4 (W4A16) quantization reduces model weights to 4-bit integers while maintaining 16-bit activations, enabling larger models to fit in GPU memory with minimal accuracy loss.
- FP8 (W8A8) quantization uses 8-bit floating point representation for weights and activations, offering a balance between memory efficiency and numerical precision for training and inference workloads.
- NVFP4 quantization uses NVIDIA’s 4-bit floating point format with two-level scaling (FP8 fine-grained scales and FP32 tensor-level scale), providing maximum memory efficiency for inference on NVIDIA Blackwell hardware.
| Architecture | Supported AI accelerators | Minimum compute capability |
|---|---|---|
| Turing | Tesla T4 | 7.5 |
| Ampere | A10, A30, A40, A100 | 8.0 |
| Ada Lovelace | L4, L40, L40S | 8.9 |
| Hopper | H100, H200, GH200 | 9.0 |
NVIDIA Blackwell architecture (B200, B300, GB200, GB300) does not support INT8 quantization in vLLM due to kernel limitations. Use FP8 or NVFP4 quantization instead.
| Architecture | Supported AI accelerators |
|---|---|
| CDNA 2 | MI210 |
| CDNA 3 | MI300X, MI325X |
| Architecture | Supported AI accelerators | Minimum compute capability |
|---|---|---|
| Ampere | A10, A30, A40, A100 | 8.0 |
| Ada Lovelace | L4, L40, L40S | 8.9 |
| Hopper | H100, H200, GH200 | 9.0 |
| Blackwell | B200, B300, GB200, GB300 | 10.0 |
NVIDIA Turing architecture (Tesla T4) does not have optimized vLLM kernel support for INT4 quantization. Use Ampere or newer architectures for INT4 inference.
| Architecture | Supported AI accelerators |
|---|---|
| CDNA 3 | MI300X, MI325X |
AMD CDNA 2 architecture (MI210) does not have optimized vLLM kernel support for INT4 quantization.
| Architecture | Supported AI accelerators | Minimum compute capability |
|---|---|---|
| Ada Lovelace | L4, L40, L40S | 8.9 |
| Hopper | H100, H200, GH200 | 9.0 |
| Blackwell | B200, B300, GB200, GB300 | 10.0 |
NVIDIA Turing architecture (Tesla T4) and Ampere architecture (A10, A30, A40, A100) AI accelerators do not support FP8 W8A8 quantization due to hardware limitations. However, FP8 weight-only (W8A16) quantization is available on these architectures by using Marlin kernels.
| Architecture | Supported AI accelerators |
|---|---|
| CDNA 3 | MI300X, MI325X |
AMD CDNA 2 architecture AI accelerators (MI210) do not support FP8 quantization due to hardware limitations.
| Architecture | Supported AI accelerators | Minimum compute capability |
|---|---|---|
| Blackwell | B200, B300, GB200, GB300 | 10.0 |
NVFP4 quantization is only available on NVIDIA Blackwell architecture AI accelerators. AMD AI accelerators do not support NVFP4 quantization.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}