Chapter 5. vLLM arguments reference

Procedure

1. Identify the vLLM arguments you want to configure. For the complete argument reference, see Red Hat AI Inference Server vLLM Server Arguments.

2. Create or edit the LLMInferenceService custom resource to include the args field under spec.template.containers for the container named main.

   Example vLLM argument configuration:

   apiVersion: serving.kserve.io/v1alpha2
   kind: LLMInferenceService
   metadata:
     name: my-vllm-service
     namespace: <my_namespace>
   spec:
     replicas: 2
     model:
       uri: hf://RedHatAI/Qwen3-8B-FP8-dynamic
       name: RedHatAI/Qwen3-8B-FP8-dynamic
     router:
       route: {}
       gateway: {}
       scheduler: {}
     template:
       containers:
       - name: main
         args:
         - --disable-uvicorn-access-log
         - --max-model-len=10000
         resources:
           limits:
             cpu: '4'
             memory: 32Gi
             nvidia.com/gpu: "1"
           requests:
             cpu: '2'
             memory: 16Gi
             nvidia.com/gpu: "1"

   where:

   • --disable-uvicorn-access-log: Disables HTTP access logs to reduce log volume.

   • --max-model-len=10000: Sets the maximum context length to 10,000 tokens.

     Note

     The arguments that you specify are merged with the default arguments provided by the system. If you specify the same argument as a default, your value takes precedence. This allows you to override specific defaults without affecting other default arguments.

3. Apply the custom resource.

   $ oc apply -f -n <my_namespace> llminferenceservice.yaml

   Example output

   llminferenceservice.serving.kserve.io/my-vllm-service configured

Verification

1. Verify that the arguments are applied correctly.

   1. Check that the LLMInferenceService is running:

      $ oc get llminferenceservice my-vllm-service -n <my_namespace>

      Example output

      NAME               READY   AGE
      my-vllm-service    True    5m

      The READY column shows True.

   2. Verify your custom arguments appear in the pod specification:

      Get the pod name:

      $ oc get pods -n <my_namespace> -l app.kubernetes.io/name=my-vllm-service

      Inspect the container arguments:

      $ oc describe pod <pod_name> -n <my_namespace> | grep -A 5 "Args:"

      Example output

          Args:
            --disable-uvicorn-access-log
            --max-model-len=10000

      Your custom arguments should appear in this list.

   3. Test that inference requests work with your configured arguments:

      Get the route URL:

      $ oc get llmisvc -n <my_namespace> my-vllm-service -o jsonpath='{.status.url}'

      Send a test request:

      $ curl -X POST https://<route_url>/v1/completions \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $(oc whoami -t)"  \
        -d { "model": "RedHatAI/Qwen3-8B-FP8-dynamic", "prompt": "Explain what Red Hat OpenShift AI is in one sentence.", "max_tokens": 100 }

      A successful response confirms that your vLLM service is running with the configured arguments.

Procedure

1. Edit the LLMInferenceService CR to override the container command.

   The following example shows how to override the startup command to exclude the --disable-uvicorn-access-log flag:

   apiVersion: serving.kserve.io/v1alpha1
   kind: LLMInferenceService
   metadata:
     name: my-vllm-service
     namespace: <namespace>
   spec:
     replicas: 2
     model:
       uri: hf://RedHatAI/Qwen3-8B-FP8-dynamic
       name: RedHatAI/Qwen3-8B-FP8-dynamic
     router:
       route: {}
       gateway: {}
       scheduler: {}
     template:
       containers:
       - name: main
         command:
         - /bin/bash
         - -c
         - |
           vllm serve "$@"
         - --
         args:
         - /mnt/models
         - --served-model-name=RedHatAI/Qwen3-8B-FP8-dynamic
         - --port=8000
         - --max-model-len=10000
         - --gpu-memory-utilization=0.9
         # Remove the following three lines if your deployment does not use TLS
         - --enable-ssl-refresh
         - --ssl-certfile=/var/run/kserve/tls/tls.crt
         - --ssl-keyfile=/var/run/kserve/tls/tls.key
         resources:
           limits:
             cpu: '4'
             memory: 32Gi
             nvidia.com/gpu: "1"
           requests:
             cpu: '2'
             memory: 16Gi
             nvidia.com/gpu: "1"

   • command: Overrides the default startup command with a minimal bash wrapper that calls vllm serve and forwards the contents of args to it as positional arguments via "$@". The -- separator after the script body is required for bash to treat subsequent entries as positional parameters rather than bash options.
   • args: Contains the vLLM server arguments. The first entry, /mnt/models, is the required model path that the storage initializer mounts into the container. The remaining entries are vLLM CLI flags. Replace the --served-model-name value with the value of spec.model.name from your LLMInferenceService.
   Note

   When you override the command field, you replace all default command-level behavior, including the --disable-uvicorn-access-log flag that is normally applied. By omitting this flag in your custom command, uvicorn access logs are enabled.

2. Apply the custom resource:

   $ oc apply -f llminferenceservice.yaml -n <namespace>

Verification

1. Verify that the LLMInferenceService is running:

   $ oc get llminferenceservice <service_name> -n <namespace>

   The output shows the service status:

   NAME              READY   AGE
   my-vllm-service   True    5m

2. View the pod logs to confirm that access log entries appear:

   $ oc logs <POD_NAME> -n <NAMESPACE> | grep "HTTP/1.1"

   • <POD_NAME> is the name of one of the vLLM pods.

   The output shows HTTP access log entries:

   INFO:     172.30.45.2:54321 - "GET /metrics HTTP/1.1" 200 OK
   INFO:     172.30.45.2:54322 - "GET /health HTTP/1.1" 200 OK
   INFO:     172.30.45.2:54323 - "POST /v1/completions HTTP/1.1" 200 OK

   Verify the log format includes the HTTP method, endpoint path, and status code.