Chapter 6. Compressing language models with Red Hat AI Model Optimization Toolkit

Procedure

1. Pull the Red Hat AI Model Optimization Toolkit container image:

   $ podman pull registry.redhat.io/rhaiis/model-opt-cuda-rhel9:3.3.0

2. Verify the LLM Compressor version installed in the container:

   $ podman run --rm -it \
     registry.redhat.io/rhaiis/model-opt-cuda-rhel9:3.3.0 \
     python -c "import llmcompressor; print(llmcompressor.__version__)"

   Example output

   v0.9.0.1

3. Create a working directory and clone the upstream LLM Compressor repository:

   $ mkdir model-opt && \
   cd model-opt && \
   git clone https://github.com/vllm-project/llm-compressor.git

4. Check out the LLM Compressor tag that matches the version that is installed in the container:

   $ cd llm-compressor && \
   git checkout v0.9.0.1

5. Create or append your HF_TOKEN Hugging Face token to the private.env file and source it:

   $ echo "export HF_TOKEN=<YOUR_HF_TOKEN>" > private.env
   $ source private.env

6. If your system has SELinux enabled, configure SELinux to allow device access:

   $ sudo setsebool -P container_use_devices 1

7. Run the llama3_example.py compression example using the Red Hat AI Inference Server model optimization container:

   $ podman run --rm \
     -v "$(pwd):/opt/app-root/model-opt" \
     --device nvidia.com/gpu=all --ipc=host \
     -e HF_TOKEN=<YOUR_HF_TOKEN> \
     registry.redhat.io/rhaiis/model-opt-cuda-rhel9:3.3.0 \
     python /opt/app-root/model-opt/llm-compressor/examples/quantization_w8a8_int8/llama3_example.py