Chapter 5. Enabling tool calling for Mistral 3 models

Procedure

1. Deploy the RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4 model with AI Inference Server and enable tool calling:

   podman run --rm -it \
     --device nvidia.com/gpu=all \
     --shm-size=4g \
     -p 8000:8000 \
     --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
     registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.3.0 \
       --model RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4 \
       --tokenizer-mode mistral \
       --config-format mistral \
       --load-format mistral \
       --enable-auto-tool-choice \
       --tool-call-parser mistral \
       --host 0.0.0.0 \
       --port 8000

   Copy to Clipboard Toggle word wrap
   • --enable-auto-tool-choice allows the server to return tool calls automatically when the model requests them.
   • --tool-call-parser mistral uses Mistral’s native tool calling format for parsing tool calls.

Verification

1. Send a chat completion request that includes tool definitions, for example:

   $ curl http://localhost:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4",
       "messages": [
         {
           "role": "user",
           "content": "What is the weather in Paris right now?"
         }
       ],
       "tools": [
         {
           "type": "function",
           "function": {
             "name": "get_weather",
             "description": "Get the current weather for a location",
             "parameters": {
               "type": "object",
               "properties": {
                 "location": {
                   "type": "string",
                   "description": "The city name"
                 }
               },
               "required": ["location"]
             }
           }
         }
       ],
       "tool_choice": "auto"
     }'

   Copy to Clipboard Toggle word wrap

   If the model decides a tool is needed, the response includes a tool_calls array instead of a final answer.

   Note

   Tool execution is performed by your application, not by the model. The model generates a structured request describing which tool to call and which arguments to use only.

2. Execute the requested tool in your application and send the tool result back to the model. For example:

   $ curl http://localhost:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "RedHatAI/Mistral-Large-3-675B-Instruct-2512-NVFP4",
       "messages": [
         {
           "role": "user",
           "content": "What is the weather in Paris right now?"
         },
         {
           "role": "assistant",
           "content": null,
           "tool_calls": [
             {
               "id": "call_123",
               "type": "function",
               "function": {
                 "name": "get_weather",
                 "arguments": "{\"location\": \"Paris\"}"
               }
             }
           ]
         },
         {
           "role": "tool",
           "tool_call_id": "call_123",
           "content": "The weather in Paris is 18°C and sunny."
         }
       ]
     }'

   Copy to Clipboard Toggle word wrap

   The model uses the tool output to generate a final natural language response and returns it as JSON.