Red Hat SummitEste contenido no está disponible en el idioma seleccionado.
Chapter 1. Red Hat Enterprise Linux AI hardware requirements
Various hardware accelerators require different requirements for serving and inferencing as well as installing, generating and training the Granite starter model on Red Hat Enterprise Linux AI.
1.1. Hardware requirements for end-to-end workflow of Granite models
The following charts show the hardware requirements for running the full InstructLab end-to-end workflow to customize the Granite student model. This includes: synthetic data generation (SDG), multi-phase training, and evaluating a custom Granite model.
1.1.1. Bare metal
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU memory | Recommended additional disk storage |
|---|---|---|---|
| NVIDIA | 2xA100 4xA100 8xA100 | 160 GB 320 GB 640 GB | 3 TB |
| NVIDIA | 2xH100 4xH100 8xH100 | 160 GB 320 GB 640 GB | 3 TB |
| NVIDIA | 2xH200 4xH200 8xH200 | 282 GB 564 GB 1128 GB | 3 TB |
| NVIDIA | 4xL40S 8xL40S | 192 GB 384 GB | 3 TB |
| AMD | 2xMI300X 4xMI300X 8xMI300X | 384 GB 768 GB 1536 GB | 3 TB |
1.1.2. IBM Cloud
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | IBM Cloud Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 2xA100 | 160 GB | gx3d-48x240x2a100p | 3 TB |
| NVIDIA | 8xH100 | 640 GB | gx3d-160x1792x8h100 | 3 TB |
| NVIDIA | 8xH200 | 1128 GB | gx3d-160x1792x8h200 | 3 TB |
| AMD | 8xMI300X | 1536 GB | gx3d-208x1792x8mi300x | 3 TB |
1.1.3. Amazon Web Services (AWS)
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | AWS Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xA100 | 320 GB | p4d.24xlarge | 3 TB |
| NVIDIA | 8xA100 | 640 GB | p4de.24xlarge | 3 TB |
| NVIDIA | 8xH100 | 640 GB | p5.48xlarge | 3 TB |
| NVIDIA | 8xL40S | 384 GB | g6e.48xlarge | 3 TB |
1.1.4. Azure
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | Azure Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xA100 | 640 GB | Standard_ND96amsr_A100_v4 | 3 TB |
| NVIDIA | 4xA100 | 320 GB | Standard_ND96asr_A100_v4 | 3 TB |
| NVIDIA | 8xH100 | 640 GB | Standard_ND96isr_H100_v5 | 3 TB |
| AMD | 8xMI300X | 1535 GB | Standard_ND96is_MI300X_v5 | 3 TB |
1.1.5. Google Cloud Platform (GCP)
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | GCP Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xA100 | 640 GB | a2-highgpu-8g | 3 TB |
| NVIDIA | 8xH100 | 640 GB | a3-highgpu-8g a3-megagpu-8g | 3 TB |
1.2. Hardware requirements for inference serving Granite models
The following charts display the minimum hardware requirements for inference serving a model on Red Hat Enterprise Linux AI.
1.2.1. Bare metal
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU memory | Recommended additional disk storage |
|---|---|---|---|
| NVIDIA | A100 | 80 GB | 1 TB |
| NVIDIA | H100 | 80 GB | 1 TB |
| NVIDIA | H200 | 141 GB | 1 TB |
| NVIDIA | GH200 (Technology Preview) | 192 GB | 1 TP |
| NVIDIA | L40S | 48 GB | 1 TB |
| NVIDIA | L4 | 24 GB | 1 TB |
| AMD | MI300X | 192 GB | 1 TB |
| Intel | Gaudi 3 (Technology Preview) | 128 GB | 1 TB |
1.2.2. Amazon Web Services (AWS)
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | AWS Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | A100 | 40 GB | P4d series | 1 TB |
| NVIDIA | H100 | 80 GB | P5 series | 1 TB |
| NVIDIA | L40S | 48 GB | G6e series | 1 TB |
| NVIDIA | L4 | 24 GB | G6 series | 1 TB |
1.2.3. IBM cloud
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | IBM Cloud Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | L4 | 24 GB | gx3 series | 1 TB |
| NVIDIA | L40S | 48 GB | gx3 series | 1 TB |
| NVIDIA | A100 | 80 GB | gx3 series | 1 TB |
| NVIDIA | H100 | 80 GB | gx3 series | 1 TB |
| NVIDIA | H200 | 141 GB | gx3 series | 1 TB |
| AMD | MI300X | 192 GB | gx3 series | 1 TB |
| Intel | Gaudi 3 (Technology Preview) | 128 GB | gx3 series | 1 TB |
1.2.4. Azure
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | Azure Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | A100 | 80 GB | ND series | 1 TB |
| NVIDIA | H100 | 80 GB | ND sereis | 1 TB |
| AMD | MI300X | 192 GB | ND series | 1 TB |
1.2.5. Google Cloud Platform (GCP)
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | GCP Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | A100 | 40 GB | A2 series | 1 TB |
| NVIDIA | H100 | 80 GB | A3 series | 1 TB |
| NVIDIA | 4xL4 | 96 GB | G2 series | 1 TB |
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}