Chapter 1. Version 3.2.5 release notes

The FlashInfer kernel sampler was disabled by default in Red Hat AI Inference Server 3.2.3 to address non-deterministic behavior and correctness errors in model output.

This change affects sampling behavior when using Flashinfer top-p and top-k sampling methods. If required, you can enable FlashInfer by setting the VLLM_USE_FLASHINFER_SAMPLER environment variable at runtime:

VLLM_USE_FLASHINFER_SAMPLER=1

VLLM_USE_FLASHINFER_SAMPLER=1

AMD ROCm AI accelerators do not support inference serving encoder-decoder models when using the vLLM v1 inference engine.

Encoder-decoder model architectures cause NotImplementedError failures with AMD ROCm accelerators. ROCm attention backends support decoder-only attention only.

Affected models include, but are not limited to, the following:

Inference fails for MP3 and M4A file formats. When querying audio models with these file formats, the system returns a "format not recognized" error.

{"error":{"message":"Error opening <_io.BytesIO object at 0x7fc052c821b0>: Format not recognised.","type":"Internal Server Error","param":null,"code":500}}

{"error":{"message":"Error opening <_io.BytesIO object at 0x7fc052c821b0>: Format not recognised.","type":"Internal Server Error","param":null,"code":500}}

This issue affects audio transcription models such as openai/whisper-large-v3 and mistralai/Voxtral-Small-24B-2507. To workaround this issue, convert audio files to WAV format before processing.

Jemalloc consumes more memory than glibc when deploying models on IBM Spyre AI accelerators.

When deploying models with jemalloc as the memory allocator, overall memory usage is significantly higher than when using glibc. In testing, jemalloc increased memory consumption by more than 50% compared to glibc. To workaround this issue, disable jemalloc by unsetting the LD_PRELOAD environment variable so the system uses glibc as the memory allocator instead.