第 5 章 查看 AI Inference 服务器指标

流程

1. 启动 AI Inference 服务器并加载您的模型，如下例所示。该命令还会公开 OpenAI 兼容 API。

   $ vllm serve unsloth/Llama-3.2-1B-Instruct

   Copy to Clipboard Toggle word wrap

2. 查询 OpenAI-compatible API 的 /metrics 端点，以获取服务器的最新指标：

   $ curl http://0.0.0.0:8000/metrics

   Copy to Clipboard Toggle word wrap

   输出示例

   # HELP vllm:iteration_tokens_total Histogram of number of tokens per engine_step.
   # TYPE vllm:iteration_tokens_total histogram
   vllm:iteration_tokens_total_sum{model_name="unsloth/Llama-3.2-1B-Instruct"} 0.0
   vllm:iteration_tokens_total_bucket{le="1.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
   vllm:iteration_tokens_total_bucket{le="8.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
   vllm:iteration_tokens_total_bucket{le="16.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
   vllm:iteration_tokens_total_bucket{le="32.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
   vllm:iteration_tokens_total_bucket{le="64.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
   vllm:iteration_tokens_total_bucket{le="128.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
   vllm:iteration_tokens_total_bucket{le="256.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
   vllm:iteration_tokens_total_bucket{le="512.0",model_name="unsloth/Llama-3.2-1B-Instruct"} 3.0
   #...

   Copy to Clipboard Toggle word wrap