Chapter 9. Tool calling through Distributed Inference with llm-d

Procedure

1. Identify the correct --tool-call-parser value for your model.

   The parser must match the chat template format used by the model. For the list of supported parsers by model, see vLLM tool calling documentation. Most supported models provide a compatible chat template, but some models also require an explicit --chat-template.

2. Create or edit the LLMInferenceService custom resource to include the tool calling arguments.

   Add --enable-auto-tool-choice and --tool-call-parser to the VLLM_ADDITIONAL_ARGS environment variable under spec.template.containers for the container named main. Save the following YAML to a file named llminferenceservice.yaml:

   apiVersion: serving.kserve.io/v1alpha2
   kind: LLMInferenceService
   metadata:
     name: my-tool-calling-service
     namespace: <my_namespace>
   spec:
     replicas: 1
     model:
       uri: hf://RedHatAI/Qwen3.5-9B-FP8-dynamic
       name: RedHatAI/Qwen3.5-9B-FP8-dynamic
     router:
       route: {}
       gateway: {}
       scheduler: {}
     template:
       containers:
       - name: main
         env:
         - name: VLLM_ADDITIONAL_ARGS
           value: >-
             --enable-auto-tool-choice
             --tool-call-parser=qwen3_coder
             --reasoning-parser=qwen3
         resources:
           limits:
             cpu: '4'
             memory: 32Gi
             nvidia.com/gpu: "1"
           requests:
             cpu: '2'
             memory: 16Gi
             nvidia.com/gpu: "1"

   where:

   --enable-auto-tool-choice

   Enables automatic tool choice for supported models. When a request uses tool_choice="auto", the model can decide whether to return a text response or generate one or more tool calls. Automatic tool choice requires a compatible --tool-call-parser for the model. This flag defaults to false in vLLM, so it must be set explicitly.

   --tool-call-parser

   Specifies the parser that vLLM uses to extract tool calls from the model’s output. This value must match the chat template format of the model.

   --reasoning-parser

   Specifies the parser that vLLM uses to separate the model’s internal reasoning from its generated content. This flag does not enable reasoning in the model but tells the engine to treat reasoning and output separately. The reasoning parser is unrelated to tool calling but is recommended for this model.

3. Apply the custom resource:

   $ oc apply -f llminferenceservice.yaml -n <my_namespace>

4. Wait for the deployment to become ready:

   $ oc get llminferenceservice my-tool-calling-service -n <my_namespace> -w

   Example output

   NAME                      READY   AGE
   my-tool-calling-service   True    3m

   The READY column shows True when the deployment is ready to serve requests.

Verification

1. Get the route URL for your model endpoint:

   $ ROUTE_URL=$(oc get llmisvc -n <my_namespace> my-tool-calling-service -o jsonpath='{.status.url}')

2. Send a tool calling request to verify the configuration:

   $ curl -s -X POST "${ROUTE_URL}/v1/chat/completions" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $(oc whoami -t)" \
     -d { "model": "RedHatAI/Qwen3.5-9B-FP8-dynamic", "temperature": 0, "messages": [ { "role": "system", "content": "You are a helpful assistant with access to tools. Use the get_weather tool when asked about weather." }, { "role": "user", "content": "What is the weather in San Francisco?" } ], "tools": [ { "type": "function", "function": { "name": "get_weather", "description": "Get the current weather in a given location.", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "City and state." }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" } | python3 -m json.tool

   The | python3 -m json.tool part of the command is optional and formats the JSON output for readability.

   Setting "tool_choice": "auto" in the request lets the model automatically decide whether to call a tool based on the request content. For the full list of tool_choice values, see Supported tool calling parameters for Distributed Inference with llm-d.

3. Verify the response contains a tool_calls array with function.name set to get_weather and function.arguments containing a location value.

   Example output

   {
       "id": "chatcmpl-abc123",
       "object": "chat.completion",
       "model": "RedHatAI/Qwen3.5-9B-FP8-dynamic",
       "choices": [
           {
               "index": 0,
               "message": {
                   "role": "assistant",
                   "content": null,
                   "tool_calls": [
                       {
                           "id": "call_abc123",
                           "type": "function",
                           "function": {
                               "name": "get_weather",
                               "arguments": "{\"location\": \"San Francisco, CA\"}"
                           }
                       }
                   ]
               },
               "finish_reason": "tool_calls"
           }
       ]
   }

   If the response contains plain text in the content field instead of a tool_calls array, see Tool calling troubleshooting for Distributed Inference with llm-d.

   To complete a multi-turn tool calling loop, send the tool execution result back to the model in a follow-up message with role set to tool. The Distributed Inference with llm-d stack preserves tool result messages in the same way as tool call requests. For the complete multi-turn pattern, see Extending Red Hat AI Inference with Tool Calling Capabilities.