Chapter 4. Validated models on Hugging Face - May 2025 collection

The following models, available from RedHat AI on Hugging Face, are validated for use with Red Hat AI Inference Server.

Expand

Table 4.1. Red Hat AI validated models - May 2025 collection
Model	Quantized variants	Hugging Face model card	Validated on
gemma-2-9b-it	FP8	Baseline FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
granite-3.1-8b-base	INT4	INT4	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
granite-3.1-8b-instruct	INT4, INT8, FP8	Baseline INT4 INT8 FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
Llama-3.1-8B-Instruct	None	Baseline	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
Llama-3.1-Nemotron-70B-Instruct-HF	FP8	Baseline FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
Llama-3.3-70B-Instruct	INT4, INT8, FP8	Baseline INT4 INT8 FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
Llama-4-Maverick-17B-128E-Instruct	FP8	Baseline FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
Llama-4-Scout-17B-16E-Instruct	INT4, FP8	Baseline INT4 FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
Meta-Llama-3.1-8B-Instruct	INT4, INT8, FP8	INT4 INT8 FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
Mistral-Small-24B-Instruct-2501	INT4, INT8, FP8	Baseline INT4 INT8 FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
Mistral-Small-3.1-24B-Instruct-2503	INT4, INT8, FP8	Baseline INT4 INT8 FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
Mixtral-8x7B-Instruct-v0.1	None	Baseline	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
phi-4	INT4, INT8, FP8	Baseline INT4 INT8 FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20
Qwen2.5-7B-Instruct	INT4, INT8, FP8	Baseline INT4 INT8 FP8	RHAIIS 3.0 RHELAI 1.5 RHOAI 2.20