Red Hat SummitChapter 3. Supported AI accelerators for Red Hat AI Inference Server
The following tables list the supported AI data center grade accelerators for Red Hat AI Inference Server 3.4.
Red Hat AI Inference Server supports data center grade AI accelerators only.
| vLLM release | AI accelerators | Requirements | vLLM architecture support | LLM Compressor support |
|---|---|---|---|---|
| vLLM v0.14.1 | NVIDIA data center GPUs:
|
|
Supported, now packaged separately in the |
Red Hat AI Inference Server 3.4.0-ea.1 is built with CUDA 13.0. The container images are backward compatible with CUDA 12.9 drivers.
If your host driver version is older than the CUDA toolkit version shipped in the AI Inference Server container, you can use NVIDIA Forward Compatibility to avoid driver upgrades.
NVIDIA T4 and A100 accelerators do not support FP8 (W8A8) quantization.
| vLLM release | AI accelerators | Requirements | vLLM architecture support | LLM Compressor support |
|---|---|---|---|---|
| vLLM v0.14.1 |
| x86 | Not supported |
AMD GPUs support FP8 (W8A8) and GGUF quantization schemes only.
| vLLM release | AI accelerators | Requirements | vLLM architecture support | LLM Compressor support |
|---|---|---|---|---|
| vLLM v0.14.1 | Google v4, v5e, v5p, v6e (Trillium) | x86 Technology Preview | Not supported |
| vLLM release | AI accelerators | Requirements | vLLM architecture support | LLM Compressor support |
|---|---|---|---|---|
| vLLM v0.14.1 | IBM Spyre for Power (ppc64le) |
| IBM Power (ppc64le) | Not supported |
| vLLM v0.14.1 | IBM Spyre for Z (s390x) |
| IBM Z (s390x) | Not supported |
| vLLM v0.14.1 | IBM AIU (x86) |
| x86 Technology Preview | Not supported |
IBM AIU support for x86 is available as a Technology Preview feature only. IBM AIU for x86 is not a Generally Available (GA) feature. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
| vLLM release | AI accelerators | Requirements | vLLM architecture support | LLM Compressor support |
|---|---|---|---|---|
| vLLM v0.14.1 | AWS Inferentia2 (Inf2), AWS Trainium (Trn1, Trn1n, Trn2) | x86 Dev Preview | Not supported |
AWS Trainium and Inferentia support is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}