Chapter 2. Version 3.4.0 release notes

Neuron variant does not support structured output with on-device sampling enabled.

When sending a chat completion request with response_format: { type: "json_schema" } to the Neuron variant, the server accepts the request but returns unstructured plain text instead of JSON. Structured output is silently ignored when on-device sampling is enabled.

To work around this issue, disable on-device sampling by passing the following configuration:

--additional-config '{"override_neuron_config": {"on_device_sampling_config": null}}'

Or in Python:

llm = LLM(
    model="my-model",
    additional_config={
        "override_neuron_config": {
            "on_device_sampling_config": None,
        }
    },
)

For more information about on-device sampling, see AWS Neuron on-device sampling documentation.

FP8-quantized models with Multi-head Latent Attention (MLA) crash on Ampere GPUs.

When serving FP8-quantized models that use MLA on GPUs with compute capability less than 8.9, the vLLM API server crashes during inference. This issue affects NVIDIA A100, A6000, and other Ampere architecture GPUs.

Affected models include RedHatAI/sarvam-105b-FP8-Dynamic and other FP8-quantized DeepSeek V2 or MLA-based models.

GPUs with compute capability 8.9 or higher, such as H100 and L40S, are not affected because they use native FP8 compute paths.

To work around this issue, use H100, L40S, or other GPUs with compute capability 8.9 or higher, or use non-FP8 quantized model variants on Ampere GPUs.

Unable to query video models deployed using Red Hat AI Inference 3.4.0.

While the model loads correctly, it fails to respond when queried. This occurs because base images do not ship the Cisco OpenH264 codec for legal and compliance reasons. The ffmpeg-free-rhai package only enables H.264 support when a compatible libopenh264.so.7 is provided at runtime. To work around this issue, provide libopenh264.so.7 at runtime. For OpenShift or Kubernetes deployments, use volume mounts to make the library available to the container.

The vllm-rocm-rhel9 container fails to start when serving mistralai/Mistral-Small-3.1-24B-Instruct-2503 on AMD ROCm GPUs.

The vLLM API server crashes during engine initialization while profiling the vision encoder with a HIP runtime error (invalid argument). This issue occurs because the model uses the Pixtral multimodal architecture, which requires precompiled GPU kernel images that are not included in Red Hat AI Inference 3.4.0. This issue will be fixed in a future z-stream release.

The vLLM API server fails to start when loading the RedHatAI/sarvam-105b-FP8-Dynamic model due to a Transformers v5 incompatibility.

During initialization, the model configuration triggers a RoPE validation error caused by a breaking API change in Transformers v5. The model’s configuration calls validate_rope(ignore_keys=…​), but Transformers v5 removed the ignore_keys parameter from RotaryEmbeddingConfigMixin.validate_rope(). This causes a runtime exception during AutoConfig.from_pretrained(), preventing model loading.

The issue occurs when trust_remote_code=True executes the model’s custom configuration directly.