Chapter 2. Version 3.4.1 release notes

The vllm-rocm-rhel9 container now starts successfully when serving mistralai/Mistral-Small-3.1-24B-Instruct-2503 on AMD ROCm GPUs.

Before this update, the vLLM API server crashed during engine initialization while profiling the vision encoder with a HIP runtime error (invalid argument). This issue is now resolved.

The vLLM API server now starts successfully when loading the RedHatAI/sarvam-105b-FP8-Dynamic model.

Before this update, during initialization, the model configuration triggered a RoPE validation error caused by a breaking API change in Transformers v5. This issue is now resolved.

Large models no longer cause out-of-memory errors during loading.

Before this update, loading large models could cause out-of-memory errors due to PyTorch memory fragmentation. With this update, PyTorch max_split_size_mb is set during model loading to prevent OOM. This issue is now resolved.

FlashInfer autotune no longer causes incorrect outputs.

Before this update, FlashInfer autotune could produce incorrect results due to correctness issues. With this update, FlashInfer autotune is temporarily disabled. This issue is now resolved.

MLA attention no longer causes CUDA illegal memory access errors.

Before this update, MLA attention could cause CUDA illegal memory access when arange_buffer was undersized. With this update, arange_buffer is sized to max_num_batched_tokens. This issue is now resolved.

WebSocket connections no longer fail with KeyError in AuthenticationMiddleware.

Before this update, WebSocket connections to the realtime API caused a KeyError on scope['method']. With this update, the middleware handles WebSocket requests correctly. This issue is now resolved.

Olmo3 reasoning parser no longer causes tokenizer errors.

Before this update, the Olmo3 reasoning parser made unnecessary tokenizer encode/decode calls. With this update, these calls are removed. This issue is now resolved.

MiniMaxM2Attention models now load correctly.

Before this update, k_norm weight sharding was incorrect in MiniMaxM2Attention. With this update, weight sharding is fixed. This issue is now resolved.

Memory pool handling no longer causes errors with expandable_segments.

Before this update, expandable_segments could conflict with the cumem memory pool. With this update, expandable_segments is auto-disabled around cumem memory pool operations. This issue is now resolved.

CUDA graphs now correctly capture max_num_batched_token.

Before this update, max_num_batched_token was not captured in CUDA graphs. With this update, the value is correctly captured. This issue is now resolved.

CUDAGraphWrapper no longer fails on gc.collect or empty_cache calls.

Before this update, the gc.collect/empty_cache patch had incorrect arity in CUDAGraphWrapper. With this update, the patch arity is corrected. This issue is now resolved.

Weight offloading now respects VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY.

Before this update, the prefetch offloader ignored the VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY environment variable. With this update, the setting is respected. This issue is now resolved.

FP8 KV cache scaling now works correctly with compressed-tensors.

Before this update, KV cache scale handling was incorrect for compressed-tensors FP8 quantization. With this update, scale handling is fixed. This issue is now resolved.

FlashInfer NVLink all2all now validates dtype correctly.

Before this update, flashinfer_nvlink_one_sided all2all accepted non-nvfp4 dtypes incorrectly. With this update, non-nvfp4 dtypes are rejected. This issue is now resolved.

Pooling models no longer experience silent weight corruption.

Before this update, buffer-reusing iterators could cause silent weight corruption in pooling models. With this update, iterators handle buffers correctly. This issue is now resolved.

NVLink transfers no longer fail due to thread-local CUDA context issues.

Before this update, thread-local CUDA context handling caused NVLink transfer failures. With this update, context handling is fixed. This issue is now resolved.

FlashInfer CUTLASS MoE no longer runs on unsupported SM121 hardware.

Before this update, FlashInfer CUTLASS MoE could run on SM121 (DGX Spark) causing failures. With this update, it is disabled on SM121. This issue is now resolved.

Chunked pipeline parallelism no longer hangs with async scheduling.

Before this update, chunked pipeline parallelism could get stuck when using async scheduling. With this update, scheduling is fixed. This issue is now resolved.

BailingMoeV2.5 MLA RoPE rotation is now correct.

Before this update, MLA RoPE did not rotate the full qk_rope_head_dim in BailingMoeV2.5 models. With this update, the full dimension is rotated. This issue is now resolved.

DeepSeek V3/R1 FP8 models now produce accurate outputs.

Before this update, the indexer did not upcast WK to BF16 for fusion, causing accuracy issues in DeepSeek V3/R1 FP8 models. With this update, upcasting is applied. This issue is now resolved.

PiecewiseBackend no longer fails with empty sym_shape_indices.

Before this update, empty sym_shape_indices caused errors in PiecewiseBackend. With this update, empty indices are handled correctly. This issue is now resolved.

MoE models using bailing_moe_linear (BailingMoeV2.5) now load and run correctly.

Before this update, incorrect RoPE dtype handling and a missing fp32 state support check caused failures in BailingMoeV2.5 models. This issue is now resolved.

FP8-quantized models with Multi-head Latent Attention (MLA) crash on Ampere GPUs.

When serving FP8-quantized models that use MLA on GPUs with compute capability less than 8.9, the vLLM API server crashes during inference. This issue affects NVIDIA A100, A6000, and other Ampere architecture GPUs.

Affected models include RedHatAI/sarvam-105b-FP8-Dynamic and other FP8-quantized DeepSeek V2 or MLA-based models.

GPUs with compute capability 8.9 or higher, such as H100 and L40S, are not affected because they use native FP8 compute paths.

To work around this issue, use H100, L40S, or other GPUs with compute capability 8.9 or higher, or use non-FP8 quantized model variants on Ampere GPUs.

Unable to query video models deployed using Red Hat AI Inference.

While the model loads correctly, it fails to respond when queried. This occurs because base images do not ship the Cisco OpenH264 codec for legal and compliance reasons. The ffmpeg-free-rhai package only enables H.264 support when a compatible libopenh264.so.7 is provided at runtime. To work around this issue, provide libopenh264.so.7 at runtime. For OpenShift or Kubernetes deployments, use volume mounts to make the library available to the container.

The Distributed Inference with llm-d EndpointPicker is bypassed when multiple HTTPRoutes share the same gateway listener.

When multiple HTTPRoutes are attached to the same wildcard Gateway listener, Istio aggregates them into a single autogenerated Gateway VirtualService and does not create the per-route ExtProcPerRoute override for the LLMInferenceService. This causes the EndpointPicker to be bypassed entirely. Requests fall back to round-robin routing; prefix cache scoring, load-aware scoring, and all intelligent scheduling are silently disabled.

This behavior is not specific to multiple LLMInferenceServices and is triggered by any HTTPRoute on the same wildcard Gateway listener, such as a token endpoint, echo service, or test route.

You can identify this issue by checking the EndpointPicker logs, which might show no per-request activity, even at verbosity levels 6 or 7. Additionally, the gateway ext_proc filter shows cluster_name: "dummy" and request_header_mode: SKIP with no per-route override applied.

This affects Istio 1.26, deployed by openshift-ingress in OSSM 3.3.x and 3.4. The upstream fix is in Istio 1.29. The following issue is related: OSSM-12585.

To work around this issue, remove or reassign any non-LLMInferenceService HTTPRoutes from the inference Gateway. Move them to a separate Gateway so the LLMInferenceService HTTPRoute is the only consumer of the wildcard listener.