Chapter 3. Red Hat Enterprise Linux AI hardware requirements
Various hardware accelerators require different requirements for serving and inferencing as well as installing, generating and training the granite-7b-starter model on Red Hat Enterprise Linux AI.
3.1. Hardware requirements for end-to-end workflow of Granite models
The following charts show the hardware requirements for running the full InstructLab end-to-end workflow to customize the Granite student model. This includes: synthetic data generation (SDG), training, and evaluating a custom Granite model.
3.1.1. Bare metal
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU memory | Recommended additional disk storage |
|---|---|---|---|
| NVIDIA | 2xA100 4xA100 8xA100 | 160 GB 320 GB 640 GB | 3 TB |
| NVIDIA | 2xH100 4xH100 8xH100 | 160 GB 320 GB 640 GB | 3 TB |
| NVIDIA | 4xL40S 8xL40S | 192 GB 384 GB | 3 TB |
| AMD (Technology preview) | 2xMI300X 4xMI300X 8xMI300X | 384 GB 768 GB 1536 GB | 3 TB |
3.1.2. IBM Cloud
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | IBM Cloud Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xH100 | 640 GB | gx3d-160x1792x8h100 | 3 TB |
3.1.3. Amazon Web Services (AWS)
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | AWS Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xA100 | 320 GB | p4d.24xlarge | 3 TB |
| NVIDIA | 8xA100 | 640 GB | p4de.24xlarge | 3 TB |
| NVIDIA | 8xH100 | 640 GB | p5.48xlarge | 3 TB |
| NVIDIA | 8xL40S | 384 GB | g6e.48xlarge | 3 TB |
3.1.4. Azure
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | Azure Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xA100 | 640 GB | Standard_ND96amsr_A100_v4 | 3 TB |
| NVIDIA | 4xA100 | 320 GB | Standard_ND96asr_A100_v4 | 3 TB |
| NVIDIA | 8xH100 | 640 GB | Standard_ND96isr_H100_v5 | 3 TB |
| AMD (Technology preview) | 8xMI300X | 1536 GB | Standard_ND96isr_MI300X_v5 | 3 TB |
3.1.5. Google Cloud Platform (GCP) (Technology preview)
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | GCP Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xA100 | 640 GB | a2-highgpu-8g | 3 TB |
| NVIDIA | 8xH100 | 640 GB | a3-highgpu-8g a3-megagpu-8g | 3 TB |
3.2. Hardware requirements for inference serving Granite models
The following charts display the minimum hardware requirements for inference serving a model on Red Hat Enterprise Linux AI.
3.2.1. Bare metal
| Hardware vendor | Supported accelerators (GPUs) | minimum Aggregate GPU memory | Recommended additional disk storage |
|---|---|---|---|
| NVIDIA | A100 | 80 GB | 1 TB |
| NVIDIA | H100 | 80 GB | 1 TB |
| NVIDIA | L40S | 48 GB | 1 TB |
| NVIDIA | L4 | 24 GB | 1 TB |
| AMD (Technology preview) | MI300X | 192 GB | 1 TB |
3.2.2. Amazon Web Services (AWS)
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | AWS Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | A100 | 40 GB | P4d series | 1 TB |
| NVIDIA | H100 | 80 GB | P5 series | 1 TB |
| NVIDIA | L40S | 48 GB | G6e series | 1 TB |
| NVIDIA | L4 | 24 GB | G6 series | 1 TB |
3.2.3. IBM cloud
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | IBM Cloud Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | L40S | 48 GB | gx series | 1 TB |
| NVIDIA | L4 | 24 GB | gx series | 1 TB |
3.2.4. Azure
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | Azure Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | A100 | 80 GB | ND series | 1 TB |
| NVIDIA | H100 | 80 GB | ND series | 1 TB |
| AMD (Technology preview) | MI300X | 192 GB | ND series | 1 TB |
3.2.5. Google Cloud Platform (GCP) (Technology preview)
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | GCP Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | A100 | 40 GB | A2 series | 1 TB |
| NVIDIA | H100 | 80 GB | A3 series | 1 TB |
| NVIDIA | 4xL4 | 96 GB | G2 series | 1 TB |
