Chapter 1. Deploying models by using Distributed Inference with llm-d

Procedure

1. Log in to the OpenShift console as a developer.

2. Create the LLMInferenceService CR with the following information:

   apiVersion: serving.kserve.io/v1alpha1
   kind: LLMInferenceService
   metadata:
     name: sample-llm-inference-service
   spec:
     replicas: 2
     model:
       uri: hf://RedHatAI/Qwen3-8B-FP8-dynamic
       name: RedHatAI/Qwen3-8B-FP8-dynamic
     router:
       route: {}
       gateway: {}
       scheduler: {}
       template:
         containers:
         - name: main
           resources:
             limits:
               cpu: '4'
               memory: 32Gi
               nvidia.com/gpu: "1"
             requests:
               cpu: '2'
               memory: 16Gi
               nvidia.com/gpu: "1"

   Customize the following parameters in the spec section of the inference service:

   • replicas - Specify the number of replicas.

   • model - Specify the URI to the model based on how the model is stored (uri) and the model name to use in chat completion requests (name).

     • S3 bucket: s3://<bucket-name>/<object-key>
     • Persistent volume claim (PVC): pvc://<claim-name>/<pvc-path>
     • OCI container image: oci://<registry_host>/<org_or_username>/<repository_name><tag_or_digest>
     • HuggingFace: hf://<model>/<optional-hash>
   • router - Provide an HTTPRoute and gateway, or leave blank to automatically create one.
3. Save the file.