Chapter 8. Inference serving with Podman on IBM Z with IBM Spyre AI accelerators

Procedure

1. Open a terminal on your server host, and log in to registry.redhat.io:

   $ podman login registry.redhat.io

   Copy to Clipboard Toggle word wrap

2. Pull the Red Hat AI Inference Server image by running the following command:

   $ podman pull registry.redhat.io/rhaiis/vllm-spyre:3.2.5

   Copy to Clipboard Toggle word wrap

3. If your system has SELinux enabled, configure SELinux to allow device access:

   $ sudo setsebool -P container_use_devices 1

   Copy to Clipboard Toggle word wrap

4. Use lspci -v to verify that the container can access the host system IBM Spyre AI accelerators:

   $ podman run -it --rm --pull=newer \
       --security-opt=label=disable \
       --device=/dev/vfio \
       --group-add keep-groups \
       --entrypoint="lspci" \
       registry.redhat.io/rhaiis/vllm-spyre:3.2.5

   Copy to Clipboard Toggle word wrap

   Example output

   0381:50:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
   0382:60:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
   0383:70:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)
   0384:80:00.0 Processing accelerators: IBM Spyre Accelerator (rev 02)

   Copy to Clipboard Toggle word wrap

5. Create a volume to mount into the container and adjust the container permissions so that the container can use it.

   $ mkdir -p ~/models && chmod g+rwX ~/models

   Copy to Clipboard Toggle word wrap
6. Download the granite-3.3-8b-instruct model into the models/ folder. See Downloading models for more information.

7. Gather the IOMMU group IDs for the available Spyre devices:

   $ lspci

   Copy to Clipboard Toggle word wrap

   Example output

   0000:00:00.0 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
   0001:00:00.0 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)
   0002:00:00.0 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev ff)
   0003:00:00.0 Processing accelerators: IBM Spyre Accelerator Virtual Function (rev 02)

   Copy to Clipboard Toggle word wrap

   Each line begins with the PCI device address, for example, 0000:00:00.0.

8. Use the PCI address to determine the IOMMU group ID for the required Spyre card, for example:

   $ readlink /sys/bus/pci/devices/<PCI_ADDRESS>/iommu_group

   Copy to Clipboard Toggle word wrap

   Example output

   ../../../kernel/iommu_groups/0

   Copy to Clipboard Toggle word wrap

   The IOMMU group ID (0) is the trailing number in the readlink output.

   Repeat for each required Spyre card.

9. Set IOMMU_GROUP_ID variables for the required Spyre cards using the readlink output. For example:

   IOMMU_GROUP_ID0=0
   IOMMU_GROUP_ID1=1
   IOMMU_GROUP_ID2=2
   IOMMU_GROUP_ID3=3

   Copy to Clipboard Toggle word wrap

10. Start the AI Inference Server container, passing in the IOMMU group ID variables for the required Spyre devices. For example, deploy the granite-3.3-8b-instruct model configured for entity extraction across 4 Spyre devices:

    podman run \
      --device /dev/vfio/vfio \
      --device /dev/vfio/${IOMMU_GROUP_ID0}:/dev/vfio/${IOMMU_GROUP_ID0}  \
      --device /dev/vfio/${IOMMU_GROUP_ID1}:/dev/vfio/${IOMMU_GROUP_ID1}  \
      --device /dev/vfio/${IOMMU_GROUP_ID2}:/dev/vfio/${IOMMU_GROUP_ID2}  \
      --device /dev/vfio/${IOMMU_GROUP_ID3}:/dev/vfio/${IOMMU_GROUP_ID3}  \
      -v $HOME/models:/models:Z \
      --pids-limit 0 \
      --userns=keep-id \
      --group-add=keep-groups \
      --memory 200G \
      --shm-size 64G \
      -p 8000:8000 \
      registry.redhat.io/rhaiis/vllm-spyre:3.2.5 \
        --model /models/granite-3.3-8b-instruct \
        -tp 4 \
        --max-model-len 32768 \
        --max-num-seqs 32

    Copy to Clipboard Toggle word wrap