Chapter 10. Serving and inferencing with Podman using CPU (x86_64 AVX2)

Procedure

1. Open a terminal on your server host, and log in to registry.redhat.io:

   $ podman login registry.redhat.io

   Copy to Clipboard Toggle word wrap

2. Pull the CPU inference image by running the following command:

   $ podman pull registry.redhat.io/rhaiis/vllm-cpu-rhel9:3.3.0

   Copy to Clipboard Toggle word wrap

3. Create a volume and mount it into the container. Adjust the container permissions so that the container can use it.

   $ mkdir -p rhaiis-cache && chmod g+rwX rhaiis-cache

   Copy to Clipboard Toggle word wrap

4. Create or append your HF_TOKEN Hugging Face token to the private.env file. Source the private.env file.

   $ echo "export HF_TOKEN=<your_HF_token>" > private.env

   Copy to Clipboard Toggle word wrap

   $ source private.env

   Copy to Clipboard Toggle word wrap

5. Verify that your CPU supports the AVX2 instruction set:

   $ grep -q avx2 /proc/cpuinfo && echo "AVX2 supported" || echo "AVX2 not supported"

   Copy to Clipboard Toggle word wrap
   Important

   If your CPU does not support AVX2, you cannot use CPU inference with Red Hat AI Inference Server.

6. Start the AI Inference Server container image.

   $ podman run --rm -it \
   --security-opt=label=disable \
   --shm-size=4g -p 8000:8000 \
   --userns=keep-id:uid=1001 \
   --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
   --env "HF_HUB_OFFLINE=0" \
   --env "VLLM_CPU_KVCACHE_SPACE=4" \
   -v ./rhaiis-cache:/opt/app-root/src/.cache:Z \
   registry.redhat.io/rhaiis/vllm-cpu-rhel9:3.3.0 \
   --model TinyLlama/TinyLlama-1.1B-Chat-v1.0

   Copy to Clipboard Toggle word wrap
   • --security-opt=label=disable: Disables SELinux label relabeling for volume mounts. Required for systems where SELinux is enabled. Without this option, the container might fail to start.
   • --shm-size=4g -p 8000:8000: Specifies the shared memory size and port mapping. Increase --shm-size to 8GB if you experience shared memory issues.
   • --userns=keep-id:uid=1001: Maps the host UID to the effective UID of the vLLM process in the container. Alternatively, you can pass --user=0, but this is less secure because it runs vLLM as root inside the container.
   • --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN": Specifies the Hugging Face API access token. Set and export HF_TOKEN with your Hugging Face token.
   • --env "VLLM_CPU_KVCACHE_SPACE=4": Allocates 4GB for the CPU key-value cache. Increase this value for larger models or longer context lengths. The default is 4GB.
   • -v ./rhaiis-cache:/opt/app-root/src/.cache:Z: Mounts the cache directory with SELinux context. The :Z suffix is required for systems where SELinux is enabled. On Debian, Ubuntu, or Docker without SELinux, omit the :Z suffix.
   • --model TinyLlama/TinyLlama-1.1B-Chat-v1.0: Specifies the Hugging Face model to serve. For CPU inference, use smaller models (under 3B parameters) for optimal performance.