Chapter 5. Troubleshooting

When you run the Red Hat AI Inference Server container image without specifying a user namespace, an unrecognized model error is returned.

podman run --rm -it \
--device nvidia.com/gpu=all \
--security-opt=label=disable \
--shm-size=4GB -p 8000:8000 \
--env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
--env "HF_HUB_OFFLINE=0" \
--env=VLLM_NO_USAGE_STATS=1 \
-v ./rhaiis-cache:/opt/app-root/src/.cache \
registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.1.0 \
--model RedHatAI/Llama-3.2-1B-Instruct-FP8

ValueError: Unrecognized model in RedHatAI/Llama-3.2-1B-Instruct-FP8. Should have a model_type key in its config.json

To resolve this error, pass --userns=keep-id:uid=1001 as a Podman parameter to ensure that the container runs with the root user.

Sometimes when Red Hat AI Inference Server downloads the model, the download fails or gets stuck. To prevent the model download from hanging, first download the model using the huggingface-cli. For example:

$ huggingface-cli download <MODEL_ID> --local-dir <DOWNLOAD_PATH>

When serving the model, pass the local model path to vLLM to prevent the model from being downloaded again.

When Red Hat AI Inference Server loads a model from disk, the process sometimes hangs. Large models consume memory, and if memory runs low, the system slows down as it swaps data between RAM and disk. Slow network file system speeds or a lack of available memory can trigger excessive swapping. This can happen in clusters where file systems are shared between cluster nodes.

Where possible, store the model in a local disk to prevent slow down during model loading. Ensure that the system has sufficient CPU memory available.

Ensure that your system has enough CPU capacity to handle the model.

Sometimes, Red Hat AI Inference Server fails to inspect the model. Errors are reported in the log. For example:

#...
  File "vllm/model_executor/models/registry.py", line xxx, in \_raise_for_unsupported
    raise ValueError(
ValueError: Model architectures [''] failed to be inspected. Please check the logs for more details.

The error occurs when vLLM fails to import the model file, which is usually related to missing dependencies or outdated binaries in the vLLM build.

Some model architectures are not supported. Refer to the list of Validated models. For example, the following errors indicate that the model you are trying to use is not supported:

Traceback (most recent call last):
#...
  File "vllm/model_executor/models/registry.py", line xxx, in inspect_model_cls
    for arch in architectures:
TypeError: 'NoneType' object is not iterable

#...
  File "vllm/model_executor/models/registry.py", line xxx, in \_raise_for_unsupported
    raise ValueError(
ValueError: Model architectures [''] are not supported for now. Supported architectures:
#...

--hf_overrides '{\"architectures\": [\"DeepseekVLV2ForCausalLM\"]}

Sometimes a runtime error occurs for certain hardware when you load 8-bit floating point (FP8) models. FP8 requires GPU hardware acceleration. Errors occur when you load FP8 models like deepseek-r1 or models tagged with the F8_E4M3 tensor type. For example:

triton.compiler.errors.CompilationError: at 1:0:
def \_per_token_group_quant_fp8(
\^
ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")
[rank0]:[W502 11:12:56.323757996 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

Sometimes when serving a model a runtime error occurs that is related to the host system. For example, you might see errors in the log like this:

INFO 05-07 19:15:17 [config.py:1901] Chunked prefill is enabled with max_num_batched_tokens=2048.
OMP: Error #179: Function Can't open SHM failed:
OMP: System error #0: Success
Traceback (most recent call last):
  File "/opt/app-root/bin/vllm", line 8, in <module>
    sys.exit(main())
..........................    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above.

You can work around this issue by passing the --shm-size=2g argument when starting vllm.

In some scenarios, the quality of the generated model responses might deteriorate after an update.

Default sampling parameters source have been updated in newer versions. For vLLM version 0.8.4 and higher, the default sampling parameters come from the generation_config.json file that is provided by the model creator. In most cases, this should lead to higher quality responses, because the model creator is likely to know which sampling parameters are best for their model. However, in some cases the defaults provided by the model creator can lead to degraded performance.

If you experience this problem, try serving the model with the old defaults by using the --generation-config vllm server argument.

You might experience a self.graph.replay() error when running a model using CUDA accelerators.

If vLLM crashes and the error trace captures the error somewhere around the self.graph.replay() method in the vllm/worker/model_runner.py module, this is most likely a CUDA error that occurs inside the CUDAGraph class.

To identify the particular CUDA operation that causes the error, add the --enforce-eager server argument to the vllm command line to disable CUDAGraph optimization and isolate the problematic CUDA operation.

You might experience accelerator and CPU communication problems that are caused by incorrect hardware or driver settings.

NVIDIA Fabric Manager is required for multi-GPU systems for some types of NVIDIA GPUs. The nvidia-fabricmanager package and associated systemd service might not be installed or the package might not be running.

Run the diagnostic Python script to check whether the NVIDIA Collective Communications Library (NCCL) and Gloo library components are communicating correctly.

On an NVIDIA system, check the fabric manager status by running the following command:

$ systemctl status nvidia-fabricmanager

On successfully configured systems, the service should be active and running with no errors.

You might experience network errors with complicated network configurations.

To troubleshoot network issues, search the logs for DEBUG statements where an incorrect IP address is listed, for example:

DEBUG 06-10 21:32:17 parallel_state.py:88] world_size=8 rank=0 local_rank=0 distributed_init_method=tcp://<incorrect_ip_address>:54641 backend=nccl

To correct the issue, set the correct IP address with the VLLM_HOST_IP environment variable, for example:

$ export VLLM_HOST_IP=<correct_ip_address>

Specify the network interface that is tied to the IP address for NCCL and Gloo:

$ export NCCL_SOCKET_IFNAME=<your_network_interface>

$ export GLOO_SOCKET_IFNAME=<your_network_interface>

You might experience Python multiprocessing warnings or runtime errors. This can be caused by code that is not properly structured for Python multiprocessing. The following is an example console warning:

WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously
    initialized. We must use the `spawn` multiprocessing start method. Setting
    VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See
    https://docs.vllm.ai/en/latest/getting_started/troubleshooting.html#python-multiprocessing
    for more information.

The following is an example Python runtime error:

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ = "__main__":
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

        To fix this issue, refer to the "Safe importing of main module"
        section in https://docs.python.org/3/library/multiprocessing.html

To resolve the runtime error, update your Python code to guard the usage of vllm behind an if__name__ = "__main__": block, for example:

if __name__ = "__main__":
    import vllm

    llm = vllm.LLM(...)

When you run the Red Hat AI Inference Server container image, sometimes it is unclear whether device pass-through errors are being caused by GPU drivers or tools such as the NVIDIA Container Toolkit.