Chapter 2. Version 3.4.0-ea.1 release notes

The FlashInfer kernel sampler was disabled by default in Red Hat AI Inference Server 3.2.3 to address non-deterministic behavior and correctness errors in model output.

This change affects sampling behavior when using FlashInfer top-p and top-k sampling methods. If required, you can enable FlashInfer by setting the VLLM_USE_FLASHINFER_SAMPLER environment variable at runtime:

VLLM_USE_FLASHINFER_SAMPLER=1

VLLM_USE_FLASHINFER_SAMPLER=1

AMD ROCm AI accelerators do not support inference serving encoder-decoder models when using the vLLM v1 inference engine.

Encoder-decoder model architectures cause NotImplementedError failures with AMD ROCm accelerators. ROCm attention backends support decoder-only attention only.

Affected models include, but are not limited to, the following:

Inference fails for MP3 and M4A file formats. When querying audio models with these file formats, the system returns a "format not recognized" error.

{"error":{"message":"Error opening <_io.BytesIO object at 0x7fc052c821b0>: Format not recognised.","type":"Internal Server Error","param":null,"code":500}}

{"error":{"message":"Error opening <_io.BytesIO object at 0x7fc052c821b0>: Format not recognised.","type":"Internal Server Error","param":null,"code":500}}

This issue affects audio transcription models such as openai/whisper-large-v3 and mistralai/Voxtral-Small-24B-2507. To work around this issue, convert audio files to WAV format before processing.

Jemalloc consumes more memory than glibc when deploying models on IBM Spyre AI accelerators.

When deploying models with jemalloc as the memory allocator, overall memory usage is significantly higher than when using glibc. In testing, jemalloc increased memory consumption by more than 50% compared to glibc. To work around this issue, disable jemalloc by unsetting the LD_PRELOAD environment variable so the system uses glibc as the memory allocator instead.

GPT-OSS model produces empty or gibberish responses when using multiple GPUs.

When deploying the GPT-OSS model with tensor parallelism greater than 1, the model produces empty or incorrect output. This issue is related to the Triton attention kernel. To work around this issue, use the --no-enable-prefix-caching CLI argument when running the model.

Google TPU inference supports only a limited set of model architectures.

Native TPU inference currently supports a subset of model architectures. Unsupported models might fail to load or might result in reduced performance. Additionally, native TPU inference support is not available for some Google models.

For the list of models with native TPU support, see vLLM TPU Recommended Models and Features.