Chapter 1. Introducing Red Hat AI

Red Hat Enterprise Linux AI (RHEL AI) empowers you to customize and contribute directly to large language models (LLMs). RHEL AI is built from the InstructLab project, which uses a fine-tuning approach called LAB (Large-Scale Alignment for Chatbots). The LAB method uses synthetic data generation (SDG) with a multi-phase training framework to produce high-quality, fine-tuned LLMs.

You can install RHEL AI as a bootable Red Hat Enterprise Linux (RHEL) container image. Each image is configured for specific hardware accelerators, including NVIDIA, AMD, and Intel, and contains various inference-serving and fine-tuning tools.

You can use your own data to create seed files, generate synthetic data, and train a Granite starter model that you can interact with and deploy.

Red Hat OpenShift AI is a comprehensive MLOps platform designed to streamline artificial intelligence and machine learning (AI/ML) development and operations across hybrid cloud environments and the edge. It fosters collaboration between data scientists and developers while ensuring IT oversight, which empowers organizations to efficiently build, train, fine-tune, and deploy predictive and generative AI models.

Offered as a self-managed or cloud service, OpenShift AI builds on the robust foundation of Red Hat OpenShift, providing a trusted platform for securely deploying AI-enabled applications and ML models at scale—across public clouds, on-premises, and edge environments.

By leveraging a broad technology ecosystem, Red Hat OpenShift AI accelerates AI/ML innovation, ensures operational consistency, enhances hybrid cloud flexibility, and upholds transparency, choice, and responsible AI practices.

Red Hat AI Inference Server provides advanced inferencing features with enterprise-grade stability and security building on the open source vLLM project.

AI Inference Server uses continuous batching and tensor parallelism to provide reduced latency and higher throughput. Continuous batching processes model requests as they arrive instead of waiting for a full batch to be accumulated. Tensor parallelism distributes LLM workloads across multiple GPUs.

To reduce the cost of inferencing models, AI Inference Server uses paged attention. LLMs use a mechanism called attention to understand conversations with users. Normally, attention uses a significant amount of memory, much of which is wasted. Paged attention addresses this memory wastage by provisioning memory for LLMs similar to the way that virtual memory works for operating systems. This approach consumes less memory, which lowers costs.

Red Hat AI Inference Server has the following features: