10.4. CUDA accelerator errors

You might experience a self.graph.replay() error when running a model using CUDA accelerators.

If vLLM crashes and the error trace captures the error somewhere around the self.graph.replay() method in the vllm/worker/model_runner.py module, this is most likely a CUDA error that occurs inside the CUDAGraph class.

To identify the particular CUDA operation that causes the error, add the --enforce-eager server argument to the vllm command line to disable CUDAGraph optimization and isolate the problematic CUDA operation.

You might experience accelerator and CPU communication problems that are caused by incorrect hardware or driver settings.

NVIDIA Fabric Manager is required for multi-GPU systems for some types of NVIDIA GPUs. The nvidia-fabricmanager package and associated systemd service might not be installed or the package might not be running.

Run the diagnostic Python script to check whether the NVIDIA Collective Communications Library (NCCL) and Gloo library components are communicating correctly.

On an NVIDIA system, check the fabric manager status by running the following command:

$ systemctl status nvidia-fabricmanager

On successfully configured systems, the service should be active and running with no errors.