Red Hat SummitChapter 1. Version 3.4 release notes
Red Hat Enterprise Linux AI is a generative AI inference platform for Linux environments that uses Red Hat AI Inference for running and optimizing models, and includes Red Hat AI Model Optimization Toolkit for model quantization, sparsity, and general compression for supported AI accelerators. Red Hat AI Model Optimization Toolkit has native Hugging Face and vLLM support. You can seamlessly integrate optimized models with deployment pipelines for faster, cost-saving inference at scale, powered by the compressed-tensors model format.
Red Hat Enterprise Linux AI is packaged as bootc container images for easy deployment on a Linux server appliances with NVIDIA CUDA, AMD ROCm, or Intel Gaudi AI accelerators installed. Download the images from registry.redhat.io.
The following bootc and disk-image container images are Generally Available (GA):
-
registry.redhat.io/rhelai3/bootc-cuda-rhel9:3.4.0 -
registry.redhat.io/rhelai3/bootc-aws-cuda-rhel9:3.4.0 -
registry.redhat.io/rhelai3/bootc-azure-cuda-rhel9:3.4.0 -
registry.redhat.io/rhelai3/bootc-gcp-cuda-rhel9:3.4.0 -
registry.redhat.io/rhelai3/bootc-rocm-rhel9:3.4.0 -
registry.redhat.io/rhelai3/bootc-azure-rocm-rhel9:3.4.0 -
registry.redhat.io/rhelai3/disk-image-cuda-rhel9:3.4.0
The following bootc container images are Technology Preview:
-
registry.redhat.io/rhelai3/bootc-gaudi-rhel9:3.4.0
There is no direct upgrade path from Red Hat Enterprise Linux AI 1.5 to Red Hat Enterprise Linux AI 3.0. You can upgrade from Red Hat Enterprise Linux AI 3.0 to 3.4 and all versions in-between.
The registry.redhat.io/rhelai3/bootc-rocm-rhel9:3.4.0 image does not include Red Hat AI Model Optimization Toolkit, which is not supported for AMD ROCm AI accelerators.
1.1. New features
Red Hat Enterprise Linux AI 3.4 packages Red Hat AI Inference 3.4, which includes the following highlights:
- Upgraded vLLM to v0.18.0
- Red Hat AI Inference 3.4 packages the upstream vLLM v0.18.0 release. You can review the complete list of updates in the upstream vLLM v0.18.0 release notes.
- Red Hat AI Model Optimization Toolkit updated to v0.10.0.1
- Red Hat AI Model Optimization Toolkit 3.4 packages the upstream LLM Compressor v0.10.0.1 release. You can review the complete list of updates in the upstream LLM Compressor v0.10.0.1 release notes.
- Systemd Quadlet service renamed from
rhaiistorhaii -
The systemd Quadlet service has been renamed from
rhaiistorhaiito align with the Red Hat AI Inference product name change. The service file is now located at/etc/containers/systemd/rhaii.container. Symbolic links from the previousrhaiispaths are maintained for backward compatibility with existing installations.
For the complete list of new features, enhancements, and known issues, see the Red Hat AI Inference 3.4 release notes.
1.2. Known issues
- NVIDIA HGX B200 and B300 servers require additional NVLink packages
The Red Hat Enterprise Linux AI bootc image does not currently include the OFED or MOFED InfiniBand drivers that NVIDIA Fabric Manager requires on HGX B200 and B300 NVSwitch-based systems. Without these drivers, Fabric Manager fails to start and GPU initialization fails with a
cudaGetDeviceCount()error (Error 802: system not yet initialized).As a workaround, you can disable NVLink by running the following command and rebooting the system:
# echo "options nvidia NVreg_NvLinkDisable=1" >> /etc/modprobe.d/nvidia.confImportantDisabling NVLink forces GPU-to-GPU communication over PCIe, which significantly reduces multi-GPU bandwidth. Expect higher latency and lower throughput for models that span multiple GPUs.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}