Chapter 9. Serving and inferencing language models with Podman using AWS Trainium and Inferentia AI accelerators

Procedure

1. Open a terminal on your AWS host, and log in to registry.redhat.io:

   $ podman login registry.redhat.io

2. Pull the Red Hat AI Inference Server image for Neuron by running the following command:

   $ podman pull registry.redhat.io/rhaii-early-access/vllm-neuron-rhel9:3.4.0-ea.1

3. Optional: Verify that the Neuron drivers and devices are available on the host.

   1. Run neuron-ls to verify that Neuron drivers are installed and to view detailed information about the Neuron hardware:

      $ neuron-ls

      Example output

      instance-type: trn1.2xlarge
      instance-id: i-0b29616c0f73dc323
      +--------+--------+----------+--------+--------------+----------+------+
      | NEURON | NEURON |  NEURON  | NEURON |     PCI      |   CPU    | NUMA |
      | DEVICE | CORES  | CORE IDS | MEMORY |     BDF      | AFFINITY | NODE |
      +--------+--------+----------+--------+--------------+----------+------+
      | 0      | 2      | 0-1      | 32 GB  | 0000:00:1e.0 | 0-7      | -1   |
      +--------+--------+----------+--------+--------------+----------+------+

      Note the number of Neuron cores available. Use this information to set --tensor-parallel-size argument when starting the container.

   2. List the Neuron devices:

      $ ls /dev/neuron*

      Example output

      /dev/neuron0

4. Create a volume for mounting into the container and adjust the permissions so that the container can use it:

   $ mkdir -p ./.cache/rhaii && chmod g+rwX ./.cache/rhaii

5. Add the HF_TOKEN Hugging Face token to the private.env file.

   $ echo "export HF_TOKEN=<huggingface_token>" > private.env

6. Append the HF_HOME variable to the private.env file.

   $ echo "export HF_HOME=./.cache/rhaii" >> private.env

   Source the private.env file.

   $ source private.env

7. Start the AI Inference Server container image:

   $ sudo podman run -it --rm \
     -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \
     -e HF_HUB_OFFLINE=0 \
     --network=host \
     --device=/dev/neuron0 \
     -p 8000:8000 \
     -v $HOME/.cache/rhaii:/root/.cache/huggingface \
     -v ./.cache/rhaii:/opt/app-root/src/.cache:Z \
     registry.redhat.io/rhaii-early-access/vllm-neuron-rhel9:3.4.0-ea.1 \
     --model Qwen/Qwen2.5-1.5B-Instruct \
     --max-model-len 4096 \
     --max-num-seqs 1 \
     --no-enable-prefix-caching \
     --port 8000 \
     --tensor-parallel-size 2 \
     --additional-config '{ "override_neuron_config": { "async_mode": false } }'

   --device=/dev/neuron0

   Map the required Neuron device. Adjust based on your model requirements and available Neuron memory.

   --no-enable-prefix-caching

   Disable prefix caching for Neuron hardware.

   --tensor-parallel-size 2

   Set --tensor-parallel-size to match the number of neuron cores being used.

   --additional-config '{ "override_neuron_config": { "async_mode": false } }'

   The --additional-config parameter passes Neuron-specific configuration. Setting async_mode to false is recommended for stability.