Chapter 2. Version 3.3.0 release notes

The FlashInfer kernel sampler was disabled by default in Red Hat AI Inference Server 3.2.3 to address non-deterministic behavior and correctness errors in model output.

This change affects sampling behavior when using FlashInfer top-p and top-k sampling methods. If required, you can enable FlashInfer by setting the VLLM_USE_FLASHINFER_SAMPLER environment variable at runtime:

VLLM_USE_FLASHINFER_SAMPLER=1

VLLM_USE_FLASHINFER_SAMPLER=1

AMD ROCm AI accelerators do not support inference serving encoder-decoder models when using the vLLM v1 inference engine.

Encoder-decoder model architectures cause NotImplementedError failures with AMD ROCm accelerators. ROCm attention backends support decoder-only attention only.

Affected models include, but are not limited to, the following:

Inference fails for MP3 and M4A file formats. When querying audio models with these file formats, the system returns a "format not recognized" error.

{"error":{"message":"Error opening <_io.BytesIO object at 0x7fc052c821b0>: Format not recognised.","type":"Internal Server Error","param":null,"code":500}}

{"error":{"message":"Error opening <_io.BytesIO object at 0x7fc052c821b0>: Format not recognised.","type":"Internal Server Error","param":null,"code":500}}

This issue affects audio transcription models such as openai/whisper-large-v3 and mistralai/Voxtral-Small-24B-2507. To work around this issue, convert audio files to WAV format before processing.

Jemalloc consumes more memory than glibc when deploying models on IBM Spyre AI accelerators.

When deploying models with jemalloc as the memory allocator, overall memory usage is significantly higher than when using glibc. In testing, jemalloc increased memory consumption by more than 50% compared to glibc. To work around this issue, disable jemalloc by unsetting the LD_PRELOAD environment variable so the system uses glibc as the memory allocator instead.

GPT-OSS model produces empty or gibberish responses when using multiple GPUs.

When deploying the GPT-OSS model with tensor parallelism greater than 1, the model produces empty or incorrect output. This issue is related to the Triton attention kernel. To work around this issue, use the --no-enable-prefix-caching CLI argument when running the model.

Google TPU inference supports only a limited set of model architectures.

Native TPU inference currently supports a subset of model architectures. For unsupported models, inference falls back to PyTorch, which may result in reduced performance. Additionally, native TPU inference support is not available for some Google models.

For the list of models with native TPU support, see vLLM TPU Installation.

Models quantized with Red Hat AI Model Optimization Toolkit may fail to load in Red Hat AI Inference Server 3.3.0.

Models generated using the registry.redhat.io/rhaiis/model-opt-cuda-rhel9:3.3.0 container image may encounter errors when loaded with the registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.3.0 container image.

To work around this issue, remove the scale_dtype and zp_dtype configuration entries from the config.json file in the quantized model’s directory.

Streaming tool calls with Mistral models return invalid JSON through the /v1/messages endpoint.

When using Mistral models through the Anthropic-compatible /v1/messages endpoint with streaming tool calls, the generated arguments for tool calls contain invalid JSON. This causes most clients to fail when parsing tool call responses.

This issue affects only the /v1/messages Anthropic-compatible endpoint with Mistral models. Mistral models using the OpenAI-compatible endpoints and non-Mistral models using the /v1/messages endpoint are not affected.