Chapter 7. vLLM arguments reference

Procedure

1. Identify the vLLM arguments you want to configure. For the complete argument reference, see Red Hat AI Inference Server vLLM Server Arguments.

2. Create or edit the LLMInferenceService custom resource to include the VLLM_ADDITIONAL_ARGS environment variable under spec.template.containers for the container named main.

   Example vLLM argument configuration:

   apiVersion: serving.kserve.io/v1alpha2
   kind: LLMInferenceService
   metadata:
     name: my-vllm-service
     namespace: <my_namespace>
   spec:
     replicas: 2
     model:
       uri: hf://RedHatAI/Qwen3-8B-FP8-dynamic
       name: RedHatAI/Qwen3-8B-FP8-dynamic
     router:
       route: {}
       gateway: {}
       scheduler: {}
     template:
       containers:
       - name: main
         env:
         - name: VLLM_ADDITIONAL_ARGS
           value: "--max-model-len=10000"
         resources:
           limits:
             cpu: '4'
             memory: 32Gi
             nvidia.com/gpu: "1"
           requests:
             cpu: '2'
             memory: 16Gi
             nvidia.com/gpu: "1"

   where:

   --max-model-len=10000

   Limits the combined prompt and output length to 10,000 tokens. If you do not specify this option, vLLM derives the maximum context length from the model configuration. Set a lower value when the model’s full context length is not required or cannot be accommodated by the available KV cache capacity. This value also affects memory and runtime structures that vLLM configures when initializing the model.

   Note

   The arguments that you specify in VLLM_ADDITIONAL_ARGS are merged with the default arguments provided by the system. If you specify the same argument as a default, your value takes precedence. You can override specific defaults without affecting other default arguments. To specify multiple arguments, use the YAML folded block scalar >- with each argument on its own line.

   You can also use VLLM_ADDITIONAL_ARGS to enable tool calling for models that support it. Add --enable-auto-tool-choice and --tool-call-parser=<parser> to the environment variable value to configure tool calling on the vLLM runtime. For the complete tool calling configuration procedure, see Configure tool calling for Distributed Inference with llm-d deployments.

3. Apply the custom resource.

   $ oc apply -f llminferenceservice.yaml -n <my_namespace>

   Example output

   llminferenceservice.serving.kserve.io/my-vllm-service configured

   If you are creating a new resource, the output shows created instead of configured.

Verification

1. Wait for the LLMInferenceService to become ready. This might take several minutes while the model downloads and loads.

   $ oc get llminferenceservice my-vllm-service -n <my_namespace> -w

   Wait until the READY column shows True before proceeding.

2. Verify that the arguments are applied correctly.

   1. Check that the LLMInferenceService is running:

      $ oc get llminferenceservice my-vllm-service -n <my_namespace>

      Example output

      NAME               READY   AGE
      my-vllm-service    True    5m

      The READY column shows True.

   2. Verify your custom arguments appear in the pod specification:

      Get the pod name:

      $ oc get pods -n <my_namespace> -l app.kubernetes.io/name=my-vllm-service

      Inspect the environment variables:

      $ oc describe pod <pod_name> -n <my_namespace> | grep -A 1 "VLLM_ADDITIONAL_ARGS"

      Example output

          VLLM_ADDITIONAL_ARGS:  --max-model-len=10000

      Your custom arguments should appear in the VLLM_ADDITIONAL_ARGS value.

   3. Test that inference requests work with your configured arguments:

      Get the route URL:

      $ ROUTE_URL=$(oc get llmisvc -n <my_namespace> my-vllm-service -o jsonpath='{.status.url}')

      Send a test request:

      $ curl -X POST "${ROUTE_URL}/v1/completions" \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $(oc whoami -t)" \
        -d { "model": "RedHatAI/Qwen3-8B-FP8-dynamic", "prompt": "Explain what Red Hat OpenShift AI is in one sentence.", "max_tokens": 100 }

      A successful response confirms that your vLLM service is running with the configured arguments.