Chapter 3. Converting models with Speculators

Procedure

1. Pull the Red Hat AI Model Optimization Toolkit container image:

   $ podman pull registry.redhat.io/rhaii-early-access/model-opt-cuda-rhel9:3.4.0-ea.2

2. Verify the Speculators version installed in the container:

   $ podman run --rm -it \
     registry.redhat.io/rhaii-early-access/model-opt-cuda-rhel9:3.4.0-ea.2 \
     pip show speculators | grep Version

   Example output

   Version: 0.4.0a1

3. Create a working directory and clone the upstream Speculators repository:

   $ mkdir model-opt && \
   cd model-opt && \
   git clone https://github.com/vllm-project/speculators.git

4. Check out the Speculators branch that matches the version installed in the container:

   $ cd speculators && \
   git checkout v0.4.0+rhaiis

5. Create or append your HF_TOKEN Hugging Face token to the private.env file and source it:

   $ echo "export HF_TOKEN=<YOUR_HF_TOKEN>" > private.env
   $ source private.env

6. If your system has SELinux enabled, configure SELinux to allow device access:

   $ sudo setsebool -P container_use_devices 1

7. Run the apply_eagle3_eagle.sh convert example using the Red Hat AI Model Optimization Toolkit container:

   $ podman run --rm \
     -v "$(pwd):/opt/app-root/model-opt" \
     --device nvidia.com/gpu=0 \
     --ipc=host \
     -e HF_TOKEN=$HF_TOKEN \
     registry.redhat.io/rhaii-early-access/model-opt-cuda-rhel9:3.4.0-ea.2 \
     bash /opt/app-root/model-opt/speculators/examples/convert/eagle3/apply_eagle3_eagle.sh

   The script downloads the Eagle 3 checkpoint, converts it to the Speculators format, and validates the result.