Chapter 10. Serve and inference language models with Podman using Intel Gaudi 3 AI accelerators

Procedure

1. Open a terminal on your Gaudi 3 server host, and log in to registry.redhat.io:

   $ podman login registry.redhat.io

2. Pull the Red Hat AI Inference Server image by running the following command:

   $ podman pull registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2

3. If your system has SELinux enabled, configure SELinux to allow device access:

   $ sudo setsebool -P container_use_devices 1

4. Optional: Verify that the Habana Processing Unit (HPU) devices are available in the host.

   1. Open a shell prompt in the Red Hat AI Inference Server container. Run the following command:

      $ podman run -it --net=host --privileged --rm --entrypoint /bin/bash registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2

   2. Verify HPU device access and basic operations by running the following Python code in the container shell prompt:

      $ python3 -c "
      import habana_frameworks.torch as ht
      import torch
      if torch.hpu.is_available():
          device_count = torch.hpu.device_count()
          print(f'HPU devices available: {device_count}')
          for i in range(device_count):
              print(f'  Device {i}: {torch.hpu.get_device_name(i)}')
          print('HPU is operational.')
      else:
          print('No HPU devices detected.')
          print('Check Gaudi SW installation and device paths.')
      "

      Example output:

      HPU devices available: 8
        Device 0: GAUDI3
        Device 1: GAUDI3
        Device 2: GAUDI3
        Device 3: GAUDI3
        Device 4: GAUDI3
        Device 5: GAUDI3
        Device 6: GAUDI3
        Device 7: GAUDI3
      HPU is operational.

   3. Optional: Verify the vllm-gaudi plugin version by running the following command in the container shell prompt:

      $ pip show vllm-gaudi | grep Version

      Example output:

      Version: 0.16.0

   4. Exit the shell prompt.

      $ exit

5. Create a local cache directory for model storage and set the permissions:

   $ mkdir -p ./.cache/rhaii && chmod g+rwX ./.cache/rhaii

6. Add the HF_TOKEN Hugging Face token to the private.env file and source it.

   $ echo "export HF_TOKEN=<huggingface_token>" > private.env

   $ source private.env

7. Start the AI Inference Server container image:

   Important

   The --privileged flag grants the container full access to host devices and capabilities. This flag is required because fine-grained device passthrough for Intel Gaudi 3 is not yet validated. For production deployments, evaluate the security implications of running privileged containers in your environment.

   $ podman run --rm -it \
     --name vllm-gaudi \
     --network=host \
     --privileged \
     --userns=keep-id:uid=1001 \
     -v /dev/shm:/dev/shm \
     -e HF_TOKEN=$HF_TOKEN \
     -e HF_HOME=/opt/app-root/src/.cache \
     -e HF_HUB_OFFLINE=0 \
     -v ./.cache/rhaii:/opt/app-root/src/.cache \
     registry.redhat.io/rhaii-early-access/vllm-gaudi-rhel9:3.4.0-ea.2 \
     --model RedHatAI/granite-4.0-h-small \
     --tensor-parallel-size 1 \
     --host 0.0.0.0 \
     --port 8000

   • --privileged: Specifies that the container has access to all HPU devices on the host. Fine-grained device passthrough for Intel Gaudi 3 is not yet validated.
   • --userns=keep-id:uid=1001: Specifies the host UID mapping to the effective UID of the vLLM process in the container. You can pass --user=0 instead, but this is less secure because it runs vLLM as root inside the container.
   • -e HF_HOME=/opt/app-root/src/.cache: Specifies the Hugging Face cache directory inside the container to match the volume mount target, ensuring that downloaded models persist across container restarts.
   • --tensor-parallel-size 1: Specifies the number of HPU devices to use for tensor parallelism. Set this value to match the number of available HPU devices for your workload.
   • --max-model-len: Optional. Specifies the maximum model context length. Defaults to the maximum context length defined by the model. If the model context length is too large for the available memory, you can reduce this value or lower the GPU memory utilization from the default of 0.9 by using the --gpu-memory-utilization argument, for example --gpu-memory-utilization=0.8.