Getting started

1. Serving and inferencing with AI Inference Server
2. Validating Red Hat AI Inference Server benefits using key metrics

Procedure

1. Run the Red Hat AI Inference Server container image with the pip list package command to view all installed Python packages. For example:

   $ podman run --rm --entrypoint=/bin/bash \
     registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.3 \
     -c "pip list"

   Copy to Clipboard Toggle word wrap

2. To view detailed information about a specific package, run the Podman command with pip show <package_name>. For example:

   $ podman run --rm --entrypoint=/bin/bash \
     registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.3 \
     -c "pip show vllm"

   Copy to Clipboard Toggle word wrap

   Example output

   Name: vllm
   Version: v0.11.0

   Copy to Clipboard Toggle word wrap

Procedure

1. Open a terminal on your server host, and log in to registry.redhat.io:

   $ podman login registry.redhat.io

   Copy to Clipboard Toggle word wrap

2. Pull the relevant the NVIDIA CUDA image by running the following command:

   $ podman pull registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.3

   Copy to Clipboard Toggle word wrap

3. If your system has SELinux enabled, configure SELinux to allow device access:

   $ sudo setsebool -P container_use_devices 1

   Copy to Clipboard Toggle word wrap

4. Create a volume and mount it into the container. Adjust the container permissions so that the container can use it.

   $ mkdir -p rhaiis-cache

   Copy to Clipboard Toggle word wrap

   $ chmod g+rwX rhaiis-cache

   Copy to Clipboard Toggle word wrap

5. Create or append your HF_TOKEN Hugging Face token to the private.env file. Source the private.env file.

   $ echo "export HF_TOKEN=<your_HF_token>" > private.env

   Copy to Clipboard Toggle word wrap

   $ source private.env

   Copy to Clipboard Toggle word wrap

6. Start the AI Inference Server container image.

   1. For NVIDIA CUDA accelerators, if the host system has multiple GPUs and uses NVSwitch, then start NVIDIA Fabric Manager. To detect if your system is using NVSwitch, first check if files are present in /proc/driver/nvidia-nvswitch/devices/, and then start NVIDIA Fabric Manager. Starting NVIDIA Fabric Manager requires root privileges.

      $ ls /proc/driver/nvidia-nvswitch/devices/

      Copy to Clipboard Toggle word wrap

      Example output

      0000:0c:09.0  0000:0c:0a.0  0000:0c:0b.0  0000:0c:0c.0  0000:0c:0d.0  0000:0c:0e.0

      Copy to Clipboard Toggle word wrap

      $ systemctl start nvidia-fabricmanager

      Copy to Clipboard Toggle word wrap
      Important

      NVIDIA Fabric Manager is only required on systems with multiple GPUs that use NVSwitch. For more information, see NVIDIA Server Architectures.

      1. Check that the Red Hat AI Inference Server container can access NVIDIA GPUs on the host by running the following command:

         $ podman run --rm -it \
         --security-opt=label=disable \
         --device nvidia.com/gpu=all \
         nvcr.io/nvidia/cuda:12.4.1-base-ubi9 \
         nvidia-smi

         Copy to Clipboard Toggle word wrap

         Example output

         +-----------------------------------------------------------------------------------------+
         | NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
         |-----------------------------------------+------------------------+----------------------+
         | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
         | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
         |                                         |                        |               MIG M. |
         |=========================================+========================+======================|
         |   0  NVIDIA A100-SXM4-80GB          Off |   00000000:08:01.0 Off |                    0 |
         | N/A   32C    P0             64W /  400W |       1MiB /  81920MiB |      0%      Default |
         |                                         |                        |             Disabled |
         +-----------------------------------------+------------------------+----------------------+
         |   1  NVIDIA A100-SXM4-80GB          Off |   00000000:08:02.0 Off |                    0 |
         | N/A   29C    P0             63W /  400W |       1MiB /  81920MiB |      0%      Default |
         |                                         |                        |             Disabled |
         +-----------------------------------------+------------------------+----------------------+
         
         +-----------------------------------------------------------------------------------------+
         | Processes:                                                                              |
         |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
         |        ID   ID                                                               Usage      |
         |=========================================================================================|
         |  No running processes found                                                             |
         +-----------------------------------------------------------------------------------------+

         Copy to Clipboard Toggle word wrap

      2. Start the container.

         $ podman run --rm -it \
         --device nvidia.com/gpu=all \
         --security-opt=label=disable \ 

         1

         
         --shm-size=4g -p 8000:8000 \ 

         2

         
         --userns=keep-id:uid=1001 \ 

         3

         
         --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \ 

         4

         
         --env "HF_HUB_OFFLINE=0" \
         --env=VLLM_NO_USAGE_STATS=1 \
         -v ./rhaiis-cache:/opt/app-root/src/.cache:Z \ 

         5

         
         registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.3 \
         --model RedHatAI/Llama-3.2-1B-Instruct-FP8 \
         --tensor-parallel-size 2 

         6

         Copy to Clipboard Toggle word wrap

         1

         Required for systems where SELinux is enabled. --security-opt=label=disable prevents SELinux from relabeling files in the volume mount. If you choose not to use this argument, your container might not successfully run.

         2

         If you experience an issue with shared memory, increase --shm-size to 8GB.

         3

         Maps the host UID to the effective UID of the vLLM process in the container. You can also pass --user=0, but this less secure than the --userns option. Setting --user=0 runs vLLM as root inside the container.

         4

         Set and export HF_TOKEN with your Hugging Face API access token

         5

         Required for systems where SELinux is enabled. On Debian or Ubuntu operating systems, or when using Docker without SELinux, the :Z suffix is not available.

         6

         Set --tensor-parallel-size to match the number of GPUs when running the AI Inference Server container on multiple GPUs.

7. In a separate tab in your terminal, make a request to your model with the API.

   curl -X POST -H "Content-Type: application/json" -d '{
       "prompt": "What is the capital of France?",
       "max_tokens": 50
   }' http://<your_server_ip>:8000/v1/completions | jq

   Copy to Clipboard Toggle word wrap

   Example output

   {
       "id": "cmpl-b84aeda1d5a4485c9cb9ed4a13072fca",
       "object": "text_completion",
       "created": 1746555421,
       "model": "RedHatAI/Llama-3.2-1B-Instruct-FP8",
       "choices": [
           {
               "index": 0,
               "text": " Paris.\nThe capital of France is Paris.",
               "logprobs": null,
               "finish_reason": "stop",
               "stop_reason": null,
               "prompt_logprobs": null
           }
       ],
       "usage": {
           "prompt_tokens": 8,
           "total_tokens": 18,
           "completion_tokens": 10,
           "prompt_tokens_details": null
       }
   }

   Copy to Clipboard Toggle word wrap

Procedure

1. Open a terminal on your server host, and log in to registry.redhat.io:

   $ podman login registry.redhat.io

   Copy to Clipboard Toggle word wrap

2. Pull the AMD ROCm image by running the following command:

   $ podman pull registry.redhat.io/rhaiis/vllm-rocm-rhel9:3.2.3

   Copy to Clipboard Toggle word wrap

3. If your system has SELinux enabled, configure SELinux to allow device access:

   $ sudo setsebool -P container_use_devices 1

   Copy to Clipboard Toggle word wrap

4. Create a volume and mount it into the container. Adjust the container permissions so that the container can use it.

   $ mkdir -p rhaiis-cache

   Copy to Clipboard Toggle word wrap

   $ chmod g+rwX rhaiis-cache

   Copy to Clipboard Toggle word wrap

5. Create or append your HF_TOKEN Hugging Face token to the private.env file. Source the private.env file.

   $ echo "export HF_TOKEN=<your_HF_token>" > private.env

   Copy to Clipboard Toggle word wrap

   $ source private.env

   Copy to Clipboard Toggle word wrap

6. Start the AI Inference Server container image.

   1. For AMD ROCm accelerators:

      1. Use amd-smi static -a to verify that the container can access the host system GPUs:

         $ podman run -ti --rm --pull=newer \
         --security-opt=label=disable \
         --device=/dev/kfd --device=/dev/dri \
         --group-add keep-groups \ 

         1

         
         --entrypoint="" \
         registry.redhat.io/rhaiis/vllm-rocm-rhel9:3.2.3 \
         amd-smi static -a

         Copy to Clipboard Toggle word wrap

         1

         You must belong to both the video and render groups on AMD systems to use the GPUs. To access GPUs, you must pass the --group-add=keep-groups supplementary groups option into the container.

      2. Start the container:

         podman run --rm -it \
         --device /dev/kfd --device /dev/dri \
         --security-opt=label=disable \ 

         1

         
         --group-add keep-groups \
         --shm-size=4GB -p 8000:8000 \ 

         2

         
         --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
         --env "HF_HUB_OFFLINE=0" \
         --env=VLLM_NO_USAGE_STATS=1 \
         -v ./rhaiis-cache:/opt/app-root/src/.cache \
         registry.redhat.io/rhaiis/vllm-rocm-rhel9:3.2.3 \
         --model RedHatAI/Llama-3.2-1B-Instruct-FP8 \
         --tensor-parallel-size 2 

         3

         Copy to Clipboard Toggle word wrap

         1

         --security-opt=label=disable prevents SELinux from relabeling files in the volume mount. If you choose not to use this argument, your container might not successfully run.

         2

         If you experience an issue with shared memory, increase --shm-size to 8GB.

         3

         Set --tensor-parallel-size to match the number of GPUs when running the AI Inference Server container on multiple GPUs.

7. In a separate tab in your terminal, make a request to the model with the API.

   curl -X POST -H "Content-Type: application/json" -d '{
       "prompt": "What is the capital of France?",
       "max_tokens": 50
   }' http://<your_server_ip>:8000/v1/completions | jq

   Copy to Clipboard Toggle word wrap

   Example output

   {
       "id": "cmpl-b84aeda1d5a4485c9cb9ed4a13072fca",
       "object": "text_completion",
       "created": 1746555421,
       "model": "RedHatAI/Llama-3.2-1B-Instruct-FP8",
       "choices": [
           {
               "index": 0,
               "text": " Paris.\nThe capital of France is Paris.",
               "logprobs": null,
               "finish_reason": "stop",
               "stop_reason": null,
               "prompt_logprobs": null
           }
       ],
       "usage": {
           "prompt_tokens": 8,
           "total_tokens": 18,
           "completion_tokens": 10,
           "prompt_tokens_details": null
       }
   }

   Copy to Clipboard Toggle word wrap

Procedure

1. Open a terminal on your TPU server host, and log in to registry.redhat.io:

   $ podman login registry.redhat.io

   Copy to Clipboard Toggle word wrap

2. Pull the Red Hat AI Inference Server image by running the following command:

   $ podman pull registry.redhat.io/rhaiis/vllm-tpu-rhel9:3.2.3

   Copy to Clipboard Toggle word wrap

3. Optional: Verify that the TPUs are available in the host.

   1. Open a shell prompt in the Red Hat AI Inference Server container. Run the following command:

      $ podman run -it --net=host --privileged -e PJRT_DEVICE=TPU --rm --entrypoint /bin/bash registry.redhat.io/rhaiis/vllm-tpu-rhel9:3.2.3

      Copy to Clipboard Toggle word wrap

   2. Verify system TPU access and basic operations by running the following Python code in the container shell prompt:

      $ python3 -c "
      import torch
      import torch_xla
      try:
          device = torch_xla.device()
          print(f'')
          print(f'XLA device available: {device}')
          x = torch.randn(3, 3).to(device)
          y = torch.randn(3, 3).to(device)
          z = torch.matmul(x, y)
          import torch_xla.core.xla_model as xm
          torch_xla.sync()
          print(f'Matrix multiplication successful')
          print(f'Result tensor shape: {z.shape}')
          print(f'Result tensor device: {z.device}')
          print(f'Result tensor: {z.data}')
          print('TPU is operational.')
      except Exception as e:
          print(f'TPU test failed: {e}')
          print('Try restarting the container to clear TPU locks')
      "

      Copy to Clipboard Toggle word wrap

      Example output

      XLA device available: xla:0
      Matrix multiplication successful
      Result tensor shape: torch.Size([3, 3])
      Result tensor device: xla:0
      Result tensor: tensor([[-1.8161,  1.6359, -3.1301],
              [-1.2205,  0.8985, -1.4422],
              [ 0.0588,  0.7693, -1.5683]], device='xla:0')
      TPU is operational.

      Copy to Clipboard Toggle word wrap

   3. Exit the shell prompt.

      $ exit

      Copy to Clipboard Toggle word wrap

4. Create a volume and mount it into the container. Adjust the container permissions so that the container can use it.

   $ mkdir ./.cache/rhaiis

   Copy to Clipboard Toggle word wrap

   $ chmod g+rwX ./.cache/rhaiis

   Copy to Clipboard Toggle word wrap

5. Add the HF_TOKEN Hugging Face token to the private.env file.

   $ echo "export HF_TOKEN=<huggingface_token>" > private.env

   Copy to Clipboard Toggle word wrap

6. Append the HF_HOME variable to the private.env file.

   $ echo "export HF_HOME=./.cache/rhaiis" >> private.env

   Copy to Clipboard Toggle word wrap

   Source the private.env file.

   $ source private.env

   Copy to Clipboard Toggle word wrap

7. Start the AI Inference Server container image:

   podman run --rm -it \
     --name vllm-tpu \
     --network=host \
     --privileged \
     --shm-size=4g \
     --device=/dev/vfio/vfio \
     --device=/dev/vfio/0 \
     -e PJRT_DEVICE=TPU \
     -e HF_HUB_OFFLINE=0 \
     -v ./.cache/rhaiis:/opt/app-root/src/.cache \
     registry.redhat.io/rhaiis/vllm-tpu-rhel9:3.2.3 \
     --model Qwen/Qwen2.5-1.5B-Instruct \
     --tensor-parallel-size 1 \ 

   1

   
     --max-model-len=256 \ 

   2

   
     --host=0.0.0.0 \
     --port=8000

   Copy to Clipboard Toggle word wrap

   1

   Set --tensor-parallel-size to match the number of TPUs.

   2

   For the best performance, set the max-model-len parameter to as low as your workload allows.

Procedure

1. On your host system, start an AI Inference Server container and serve a model.

   $ podman run --rm -it --device nvidia.com/gpu=all \
   --shm-size=4GB -p 8000:8000 \
   --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
   --env "HF_HUB_OFFLINE=0" \
   -v ./rhaiis-cache:/opt/app-root/src/.cache \
   --security-opt=label=disable \
   registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.3 \
   --model RedHatAI/Llama-3.2-1B-Instruct-FP8

   Copy to Clipboard Toggle word wrap

2. In a separate terminal tab, install the benchmark tool dependencies.

   $ pip install vllm pandas datasets

   Copy to Clipboard Toggle word wrap

3. Clone the vLLM Git repository:

   $ git clone https://github.com/vllm-project/vllm.git

   Copy to Clipboard Toggle word wrap

4. Run the ./vllm/benchmarks/benchmark_serving.py script.

   $ python vllm/benchmarks/benchmark_serving.py --backend vllm --model RedHatAI/Llama-3.2-1B-Instruct-FP8 --num-prompts 100 --dataset-name random  --random-input 1024 --random-output 512 --port 8000

   Copy to Clipboard Toggle word wrap

Procedure

1. Open a terminal and log in to registry.redhat.io:

   $ podman login registry.redhat.io

   Copy to Clipboard Toggle word wrap

2. Pull the specific Red Hat AI Inference Server container image for the AI accelerator that is installed. For example, to pull the Red Hat AI Inference Server container for Google cloud TPUs, run the following command:

   $ podman pull registry.redhat.io/rhaiis/vllm-tpu-rhel9:3.2.3

   Copy to Clipboard Toggle word wrap

3. Run the collect environment script in the container:

   $ podman run --rm -it \
     --name vllm-tpu \
     --network=host \
     --privileged \
     --device=/dev/vfio/vfio \
     --device=/dev/vfio/0 \
     -e PJRT_DEVICE=TPU \
     -e HF_HUB_OFFLINE=0 \
     -v ./.cache/rhaiis:/opt/app-root/src/.cache:Z \
     --entrypoint vllm collect-env \
     registry.redhat.io/rhaiis/vllm-tpu-rhel9:3.2.3

   Copy to Clipboard Toggle word wrap