Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 3. Downloading Large Language models

Red Hat Enterprise Linux AI allows you to customize or chat with various Large Language Models (LLMs) provided and built by Red Hat and IBM. You can download these models from the Red Hat RHEL AI registry. You can upload any custom model to an S3 bucket.

Table 3.1. Red Hat Enterprise Linux AI version 1.5 LLMs
Large Language Models (LLMs)TypeSizePurposeModel familyNVIDIA Accelerator SupportAMD Accelerator SupportIntel Accelerator Support

granite-3.1-8b-starter-v2.1

LAB fine-tuned granite starter model

16.0 GB

Version 2 of the default Granite 3.1 base model for customizing and fine-tuning

Granite 3.1

Generally Available

Generally Available

Not Available

granite-3.1-8b-lab-v2.1

LAB fine-tuned granite model

16.0 GB

Version 2 of the default Granite 3.1 model for inference serving

Granite 3.1

Generally Available

Generally Available

Not Available

granite-3.1-8b-starter-v2

LAB fine-tuned granite starter model

16.0 GB

Version 2 of the default Granite 3.1 base model for customizing and fine-tuning

Granite 3.1

Not Available

Not Available

Technology Preview

granite-3.1-8b-lab-v2

LAB fine-tuned granite model

16.0 GB

Version 2 of the default Granite 3.1 model for inference serving

Granite 3.1

Not Available

Not Available

Technology Preview

granite-8b-code-instruct

LAB fine-tuned granite code model

15.0 GB

LAB fine-tuned granite code model for inference serving

Granite Code models

Technology Preview

Technology Preview

Technology Preview

granite-8b-code-base

Granite fine-tuned code model

15.0 GB

Granite code model for inference serving

Granite Code models

Technology Preview

Technology Preview

Technology Preview

mixtral-8x7b-instruct-v0-1

Default teacher model

87.0 GB

Default teacher model for running Synthetic data generation (SDG)

Mixtral

Generally Available

Generally Available

Technology Preview

llama-3.3-70b-Instruct

Optional teacher model

74.0 GB

Optional teacher model for running Synthetic data generation (SDG)

Llama

Technology Preview

Not Available

Not Available

prometheus-8x7b-v2-0

Evaluation judge model

87.0 GB

Judge model for multi-phase training and evaluation

Prometheus 2

Generally Available

Generally Available

Technology Preview

Important

Using the granite-8b-code-instruct or granite-8b-code-base Large Language models (LLMs) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Models required for customizing the Granite LLM

  • The granite-7b-starter or granite-8b-starter-v1 base LLM depending on your hardware vendor.
  • The mixtral-8x7b-instruct-v0-1 teacher model for SDG.
  • The prometheus-8x7b-v2-0 judge model for training and evaluation.

Additional tools required for customizing an LLM

The Low-rank adaptation (LoRA) adaptors enhance the efficiency of the Synthetic Data Generation (SDG) process.

  • The skills-adapter-v3 LoRA layered skills adapter for SDG.

  • The knowledge-adapter-v3 LoRA layered knowledge adapter for SDG.

    Example command for downloading the adaptors

    $ ilab model download --repository docker://registry.redhat.io/rhelai1/knowledge-adapter-v3 --release latest
    Copy to Clipboard

Important

The LoRA layered adapters do not show up in the output of the ilab model list command. You can see the skills-adapter-v3 and knowledge-adapter-v3 files in the ls ~/.cache/instructlab/models folder.

3.1. Downloading the models from a Red Hat repository

You can download the additional optional models created by Red Hat and IBM.

Prerequisites

  • You installed RHEL AI with the bootable container image.
  • You initialized InstructLab.
  • You created a Red Hat registry account and logged in on your machine.
  • You have root user access on your machine.

Procedure

  1. To download the additional LLM models, run the following command: 

    $ ilab model download --repository docker://<repository_and_model> --release <release>
    Copy to Clipboard

    where:

    <repository_and_model>
    Specifies the repository location of the model as well as the model. You can access the models from the registry.redhat.io/rhelai1/ repository.
    <release>
    Specifies the version of the model. Set to 1.5 for the models that are supported on RHEL AI version 1.5. Set to latest for the latest version of the model.

    Example command

    $ ilab model download --repository docker://registry.redhat.io/rhelai1/granite-3.1-8b-starter-v1 --release latest
    Copy to Clipboard

Verification

  1. You can view all the downloaded models, including the new models after training, on your system with the following command: 

    $ ilab model list
    Copy to Clipboard

    Example output

    +-----------------------------------+---------------------+---------+
| Model Name                        | Last Modified       | Size    |
+-----------------------------------+---------------------+---------+
| models/prometheus-8x7b-v2-0       | 2024-08-09 13:28:50 |  87.0 GB|
| models/mixtral-8x7b-instruct-v0-1 | 2024-08-09 13:28:24 |  87.0 GB|
| models/granite-3.1-8b-starter-v1  | 2024-08-09 14:28:40 |  16.6 GB|
| models/granite-3.1-8b-lab-v1      | 2024-08-09 14:40:35 |  16.6 GB|
+-----------------------------------+---------------------+---------+
    Copy to Clipboard

  2. You can also list the downloaded models in the ls ~/.cache/instructlab/models folder by running the following command: 

    $ ls ~/.cache/instructlab/models
    Copy to Clipboard

    Example output

    granite-3.1-8b-starter-v1
granite-3.1-8b-lab-v1
    Copy to Clipboard

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat