이 콘텐츠는 선택한 언어로 제공되지 않습니다.

Chapter 3. Downloading Large Language models

Red Hat Enterprise Linux AI allows you to customize or chat with various Large Language Models (LLMs) provided and built by Red Hat and IBM. You can download these models from the Red Hat RHEL AI registry. You can upload any custom model to an S3 bucket.

Expand

Table 3.1. Red Hat Enterprise Linux AI version 1.5 LLMs
Large Language Models (LLMs)	Type	Size	Purpose	Model family	NVIDIA Accelerator Support	AMD Accelerator Support	Intel Accelerator Support
`granite-3.1-8b-starter-v2.1`	LAB fine-tuned granite starter model	16.0 GB	Version 2 of the default Granite 3.1 base model for customizing and fine-tuning	Granite 3.1	Generally Available	Generally Available	Not Available
`granite-3.1-8b-lab-v2.1`	LAB fine-tuned granite model	16.0 GB	Version 2 of the default Granite 3.1 model for inference serving	Granite 3.1	Generally Available	Generally Available	Not Available
`granite-3.1-8b-starter-v2`	LAB fine-tuned granite starter model	16.0 GB	Version 2 of the default Granite 3.1 base model for customizing and fine-tuning	Granite 3.1	Not Available	Not Available	Technology Preview
`granite-3.1-8b-lab-v2`	LAB fine-tuned granite model	16.0 GB	Version 2 of the default Granite 3.1 model for inference serving	Granite 3.1	Not Available	Not Available	Technology Preview
`granite-8b-code-instruct`	LAB fine-tuned granite code model	15.0 GB	LAB fine-tuned granite code model for inference serving	Granite Code models	Technology Preview	Technology Preview	Technology Preview
`granite-8b-code-base`	Granite fine-tuned code model	15.0 GB	Granite code model for inference serving	Granite Code models	Technology Preview	Technology Preview	Technology Preview
`mixtral-8x7b-instruct-v0-1`	Default teacher model	87.0 GB	Default teacher model for running Synthetic data generation (SDG)	Mixtral	Generally Available	Generally Available	Technology Preview
`llama-3.3-70b-Instruct`	Optional teacher model	74.0 GB	Optional teacher model for running Synthetic data generation (SDG)	Llama	Technology Preview	Not Available	Not Available
`prometheus-8x7b-v2-0`	Evaluation judge model	87.0 GB	Judge model for multi-phase training and evaluation	Prometheus 2	Generally Available	Generally Available	Technology Preview

Important

Using the granite-8b-code-instruct or granite-8b-code-base Large Language models (LLMs) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Models required for customizing the Granite LLM

The granite-7b-starter or granite-8b-starter-v1 base LLM depending on your hardware vendor.
The mixtral-8x7b-instruct-v0-1 teacher model for SDG.
The prometheus-8x7b-v2-0 judge model for training and evaluation.

Additional tools required for customizing an LLM

The Low-rank adaptation (LoRA) adaptors enhance the efficiency of the Synthetic Data Generation (SDG) process.

The skills-adapter-v3 LoRA layered skills adapter for SDG.

The knowledge-adapter-v3 LoRA layered knowledge adapter for SDG.

Example command for downloading the adaptors

ilab model download --repository docker://registry.redhat.io/rhelai1/knowledge-adapter-v3 --release latest

$ ilab model download --repository docker://registry.redhat.io/rhelai1/knowledge-adapter-v3 --release latest

Copy to Clipboard

Toggle word wrap

Important

The LoRA layered adapters do not show up in the output of the ilab model list command. You can see the skills-adapter-v3 and knowledge-adapter-v3 files in the ls ~/.cache/instructlab/models folder.

3.1. Downloading the models from a Red Hat repository
링크 복사

You can download the additional optional models created by Red Hat and IBM.

Prerequisites

You installed RHEL AI with the bootable container image.
You initialized InstructLab.
You created a Red Hat registry account and logged in on your machine.
You have root user access on your machine.

Procedure

To download the additional LLM models, run the following command:
```
ilab model download --repository docker://<repository_and_model> --release <release>
```
```
$ ilab model download --repository docker://<repository_and_model> --release <release>
```
Copy to Clipboard Toggle word wrap
where:
<repository_and_model>
Specifies the repository location of the model as well as the model. You can access the models from the registry.redhat.io/rhelai1/ repository.
<release>
Specifies the version of the model. Set to 1.5 for the models that are supported on RHEL AI version 1.5. Set to latest for the latest version of the model.
Example command
```
ilab model download --repository docker://registry.redhat.io/rhelai1/granite-3.1-8b-starter-v1 --release latest
```
```
$ ilab model download --repository docker://registry.redhat.io/rhelai1/granite-3.1-8b-starter-v1 --release latest
```
Copy to Clipboard Toggle word wrap

Verification

You can view all the downloaded models, including the new models after training, on your system with the following command:

ilab model list

$ ilab model list

Copy to Clipboard

Toggle word wrap

Example output

+-----------------------------------+---------------------+---------+
| Model Name                        | Last Modified       | Size    |
+-----------------------------------+---------------------+---------+
| models/prometheus-8x7b-v2-0       | 2024-08-09 13:28:50 |  87.0 GB|
| models/mixtral-8x7b-instruct-v0-1 | 2024-08-09 13:28:24 |  87.0 GB|
| models/granite-3.1-8b-starter-v1  | 2024-08-09 14:28:40 |  16.6 GB|
| models/granite-3.1-8b-lab-v1      | 2024-08-09 14:40:35 |  16.6 GB|
+-----------------------------------+---------------------+---------+

+-----------------------------------+---------------------+---------+
| Model Name                        | Last Modified       | Size    |
+-----------------------------------+---------------------+---------+
| models/prometheus-8x7b-v2-0       | 2024-08-09 13:28:50 |  87.0 GB|
| models/mixtral-8x7b-instruct-v0-1 | 2024-08-09 13:28:24 |  87.0 GB|
| models/granite-3.1-8b-starter-v1  | 2024-08-09 14:28:40 |  16.6 GB|
| models/granite-3.1-8b-lab-v1      | 2024-08-09 14:40:35 |  16.6 GB|
+-----------------------------------+---------------------+---------+

Copy to Clipboard

Toggle word wrap

You can also list the downloaded models in the ls ~/.cache/instructlab/models folder by running the following command:
```
ls ~/.cache/instructlab/models
```
```
$ ls ~/.cache/instructlab/models
```
Copy to Clipboard Toggle word wrap
Example output
```
granite-3.1-8b-starter-v1
granite-3.1-8b-lab-v1
```
```
granite-3.1-8b-starter-v1
granite-3.1-8b-lab-v1
```
Copy to Clipboard Toggle word wrap

이 콘텐츠는 선택한 언어로 제공되지 않습니다.

Chapter 3. Downloading Large Language models

3.1. Downloading the models from a Red Hat repository
링크 복사

자세한 정보

평가판, 구매 및 판매

커뮤니티

Red Hat 문서 정보

보다 포괄적 수용을 위한 오픈 소스 용어 교체

Red Hat 소개

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

이 콘텐츠는 선택한 언어로 제공되지 않습니다.

Chapter 3. Downloading Large Language models

3.1. Downloading the models from a Red Hat repository링크 복사링크가 클립보드에 복사되었습니다!

자세한 정보

평가판, 구매 및 판매

커뮤니티

Red Hat 문서 정보

보다 포괄적 수용을 위한 오픈 소스 용어 교체

Red Hat 소개

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

3.1. Downloading the models from a Red Hat repository
링크 복사