Chapter 1. Red Hat Enterprise Linux AI 1.3 release notes

Red Hat Enterprise Linux AI is installable as a bootable image. This image contains various tooling for interacting with RHEL AI. The image includes: Red Hat Enterprise Linux 9.4, Python version 3.11 and InstructLab tools for model fine-tuning. For more information about installing Red Hat Enterprise Linux AI, see Installation overview and the "Installation feature tracker"

After installing Red Hat Enterprise Linux AI, you can set up your RHEL AI environment with the InstructLab tools.

On Red Hat Enterprise Linux AI, you can customize your taxonomy tree using custom YAML files so a model can learn domain-specific information. You host your knowledge data in a Git repository and fine-tune a model with that data. For detailed documentation on how to create a knowledge markdown and YAML file, see Customizing your taxonomy tree.

You can use Red Hat Enterprise Linux AI to customize a granite starter LLM with your domain specific skills and knowledge. RHEL AI includes the LAB enhanced method of Synthetic Data Generation (SDG) and multi-phase training.

The 1.3 version of RHEL AI sometimes auto-detects the incorrect system profile on machines with AMD accelerators.

You can select the correct system profile on AMD with the following command:

$ ilab config init --profile ~/.local/share/instructlab/internal/system_profiles/amd/mi300x/<specified-profile>

After installing Red Hat Enterprise Linux AI using the ISO image or upgrading to a system using the bootc-amd-rhel9 container, the amd-smi tool does not work by default. To enable amd-smi, add the proper ROCm version to your user PATH variable with the following command:

$ export PATH="$PATH:/opt/rocm-6.1.2/bin"

The default max_batch_len parameter in the 4xL40s configuration needs to be updated in order for training to run properly.

You can edit the config.yaml file with the following command:

$ ilab config edit

Edit the config.yaml file to match the following parameter:

train:
    max_batch_len: 15000

$ sudo rpm --restore shadow-utils

RHEL AI sometimes auto-detects the incorrect system profile on machines with H100 or A100 accelerators.

You can select the correct profile by re-initializing and passing the correct system profile.

$ ilab config init --profile <path-to-system-profile>

$ ilab config init --profile ~/.local/share/instructlab/internal/system_profiles/nvidia/h100/h100.yaml

You need to change the default gpu count in the Gaudi3 configuration in order to inference serve on your machine.

You can edit the config.yaml file with the following command:

$ ilab config edit

Edit the config.yaml so any gpu parameter is set to the value 1.

evaluate:
    gpus: 1
generate:
    vllm:
        gpus: 1
serve:
    vllm:
        gpus: 1

After installing Red Hat Enterprise Linux AI on NVIDIA systems, you may see the following error when serving or training a model.

INFO 2024-11-26 22:18:04,244 instructlab.model.serve_backend:56: Using model '/var/home/cloud-user/.local/share/instructlab/phased/phase2/checkpoints/hf_format/samples_29117' with -1 gpu-lay
ers and 4096 max context size.
INFO 2024-11-26 22:18:04,244 instructlab.model.serve_backend:88: '--gpus' flag used alongside '--tensor-parallel-size' in the vllm_args section of the config file. Using value of the --gpus

File "/opt/app-root/lib64/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 105, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib64/python3.11/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/lib64/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 192, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start

To resolve this issue, you need the run the following commands:

$ sudo systemctl stop nvidia-persistenced.service
$ sudo systemctl start nvidia-fabricmanager.service
$ sudo systemctl start nvidia-persistenced.service

Red Hat Enterprise Linux AI version 1.3 currently does not support graphical based installation with the technology previewed AMD ISOs. Ensure that the text parameter in your kickstart file is configured for non-interactive installs. You can also pass inst.text in your shell during interactive installation to avoid an install time crash.

For SDG to run on 4xL40s, you need to run SDG with the --num-cpus flag and set to the value of 4.

$ ilab data generate --num-cpus 4

When evaluating a model built from the granite-8b-starter-v1 LLM, there might an error where vLLM does not start when running the MMLU and MMLU_BRANCH benchmarks.

If vLLM does not start, add the following parameter to the serve section of your config.yaml file:

serve:
  vllm:
    vllm_args: [--dtype bfloat16]

Red Hat Enterprise Linux AI version 1.3 does not support kdump over nfs without configuration. To use this feature, run the following commands:

mkdir -p /var/lib/kdump/dracut.conf.d
echo "dracutmodules=''" > /var/lib/kdump/dracut.conf.d/99-kdump.conf
echo "omit_dracutmodules=''" >> /var/lib/kdump/dracut.conf.d/99-kdump.conf
echo "dracut_args --confdir /var/lib/kdump/dracut.conf.d --install /usr/lib/passwd --install /usr/lib/group" >> /etc/kdump.conf
systemctl restart kdump