Chapter 3. Red Hat Enterprise Linux AI hardware requirements
Various hardware accelerators require different requirements for serving and inferencing as well as installing, generating and training the granite-7b-starter model on Red Hat Enterprise Linux AI.
3.1. Hardware requirements for end-to-end workflow of Granite models
The following charts show the hardware requirements for running the full InstructLab end-to-end workflow to customize the Granite student model. This includes: synthetic data generation (SDG), training, and evaluating a custom Granite model.
3.1.1. Bare metal
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU memory | Recommended additional disk storage |
|---|---|---|---|
| NVIDIA | 2xA100 4xA100 8xA100 | 160 GB 320 GB 640 GB | 1 TB |
| NVIDIA | 2xH100 4xH100 8xH100 | 160 GB 320 GB 640 GB | 1 TB |
| NVIDIA | 4xL40S 8xL40S | 192 GB 384 GB | 1 TB |
3.1.2. Amazon Web Services (AWS)
| Hardware vendor | Supported accelerators (GPUs) | Aggregate GPU Memory | AWS Instance | Recommended additional disk storage |
|---|---|---|---|---|
| NVIDIA | 8xA100 | 640 GB | p4de.24xlarge | 1 TB |
| NVIDIA | 8xH100 | 640 GB | p5.48xlarge | 1 TB |
3.2. Hardware requirements for inference serving Granite models
The following charts display the minimum hardware requirements for inference serving a model on Red Hat Enterprise Linux AI.
3.2.1. Bare metal
| Hardware vendor | Supported accelerators (GPUs) | minimum Aggregate GPU memory | Recommended additional disk storage |
|---|---|---|---|
| NVIDIA | A100 | 80 GB | 1 TB |
| NVIDIA | H100 | 80 GB | 1 TB |
| NVIDIA | L40S | 48 GB | 1 TB |
| NVIDIA | L4 | 24 GB | 1 TB |
3.2.2. Amazon Web Services (AWS)
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | Recommended additional disk storage |
|---|---|---|---|
| NVIDIA | A100 | 80 GB | 1 TB |
| NVIDIA | H100 | 80 GB | 1 TB |
| NVIDIA | L40S | 48 GB | 1 TB |
| NVIDIA | L4 | 24 GB | 1 TB |
3.2.3. IBM cloud
| Hardware vendor | Supported accelerators (GPUs) | Minimum Aggregate GPU Memory | Recommended additional disk storage |
|---|---|---|---|
| NVIDIA | L40S | 48 GB | 1 TB |
| NVIDIA | L4 | 24 GB | 1 TB |
