이 콘텐츠는 선택한 언어로 제공되지 않습니다.
Chapter 3. Red Hat Enterprise Linux AI hardware requirements
Various hardware accelerators require different requirements for serving and inferencing as well as installing, generating and training the granite-7b-starter model on Red Hat Enterprise Linux AI.
3.1. Hardware requirements for end-to-end workflow of Granite models
The following charts show the hardware requirements for running the full InstructLab end-to-end workflow to customize the Granite student model. This includes: synthetic data generation (SDG), training, and evaluating a custom Granite model.
3.1.1. Bare metal
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU memory | Recommended additional disk storage |
|---|---|---|---|
| NVIDIA | 2xA100 4xA100 8xA100 | 160 GB 320 GB 640 GB | 3 TB |
| NVIDIA | 2xH100 4xH100 8xH100 | 160 GB 320 GB 640 GB | 3 TB |
| NVIDIA | 4xL40S 8xL40S | 192 GB 384 GB | 3 TB |
| AMD (Technology preview) | 2xMI300X 4xMI300X 8xMI300X | 384 GB 768 GB 1536 GB | 3 TB |
3.1.2. IBM Cloud
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | IBM Cloud Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xH100 | 640 GB | gx3d-160x1792x8h100 | 3 TB |
3.1.3. Amazon Web Services (AWS)
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | AWS Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xA100 | 320 GB | p4d.24xlarge | 3 TB |
| NVIDIA | 8xA100 | 640 GB | p4de.24xlarge | 3 TB |
| NVIDIA | 8xH100 | 640 GB | p5.48xlarge | 3 TB |
| NVIDIA | 8xL40S | 384 GB | g6e.48xlarge | 3 TB |
3.1.4. Azure
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | Azure Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xA100 | 640 GB | Standard_ND96amsr_A100_v4 | 3 TB |
| NVIDIA | 4xA100 | 320 GB | Standard_ND96asr_A100_v4 | 3 TB |
| NVIDIA | 8xH100 | 640 GB | Standard_ND96isr_H100_v5 | 3 TB |
| AMD (Technology preview) | 8xMI300X | 1536 GB | Standard_ND96isr_MI300X_v5 | 3 TB |
3.1.5. Google Cloud Platform (GCP) (Technology preview)
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | GCP Instances | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xA100 | 640 GB | a2-highgpu-8g | 3 TB |
| NVIDIA | 8xH100 | 640 GB | a3-highgpu-8g a3-megagpu-8g | 3 TB |
3.2. Hardware requirements for inference serving Granite models
The following charts display the minimum hardware requirements for inference serving a model on Red Hat Enterprise Linux AI.
3.2.1. Bare metal
| Hardware vendor | Supported accelerators (GPUs) | minimum Aggregate GPU memory | Recommended additional disk storage |
|---|---|---|---|
| NVIDIA | A100 | 80 GB | 1 TB |
| NVIDIA | H100 | 80 GB | 1 TB |
| NVIDIA | L40S | 48 GB | 1 TB |
| NVIDIA | L4 | 24 GB | 1 TB |
| AMD (Technology preview) | MI300X | 192 GB | 1 TB |
3.2.2. Amazon Web Services (AWS)
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | AWS Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | A100 | 40 GB | P4d series | 1 TB |
| NVIDIA | H100 | 80 GB | P5 series | 1 TB |
| NVIDIA | L40S | 48 GB | G6e series | 1 TB |
| NVIDIA | L4 | 24 GB | G6 series | 1 TB |
3.2.3. IBM cloud
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | IBM Cloud Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | L40S | 48 GB | gx series | 1 TB |
| NVIDIA | L4 | 24 GB | gx series | 1 TB |
3.2.4. Azure
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | Azure Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | A100 | 80 GB | ND series | 1 TB |
| NVIDIA | H100 | 80 GB | ND series | 1 TB |
| AMD (Technology preview) | MI300X | 192 GB | ND series | 1 TB |
3.2.5. Google Cloud Platform (GCP) (Technology preview)
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | GCP Instance family | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | A100 | 40 GB | A2 series | 1 TB |
| NVIDIA | H100 | 80 GB | A3 series | 1 TB |
| NVIDIA | 4xL4 | 96 GB | G2 series | 1 TB |
