Red Hat SummitEste conteúdo não está disponível no idioma selecionado.
Chapter 2. Model serving options for RHEL AI
When deploying models with Red Hat Enterprise Linux AI and Red Hat AI Inference Server, you can choose from different approaches to supply a model for inference serving. Understanding the differences between each approach helps you select the right one for your deployment scenario.
2.1. Hugging Face models
Hugging Face models are the recommended approach for RHEL AI deployments that use the Red Hat AI Inference Server systemd Quadlet service. With this approach, you can either download models directly from Hugging Face Hub at runtime, or pre-download models to the local file system for offline serving.
Use Hugging Face models in the following scenarios:
- You are deploying RHEL AI using the Red Hat AI Inference Server systemd Quadlet service.
- You want to serve models from Hugging Face Hub (online mode) or from a local directory (offline mode).
- You want to use Red Hat AI Model Optimization Toolkit to quantize and compress models before serving.
2.2. ModelCar container images
ModelCar container images are OCI-compliant container images that package language models as standard container images. You can pull ModelCar images from registry.redhat.io using podman pull and mount them directly into the Red Hat AI Inference Server vLLM container by using the --mount type=image option.
For a list of available ModelCar container images, see ModelCar container images.
Use ModelCar container images in the following scenarios:
- You are running Red Hat AI Inference Server with Podman directly, outside of the RHEL AI Quadlet service.
- You want to package and distribute models as container images.
- You want a container-native workflow for model distribution and versioning.
For example, to pull and run a ModelCar image with Podman:
Pull the ModelCar image:
$ podman pull registry.redhat.io/rhelai1/modelcar-granite-8b-code-instruct:1.4Inference serve the ModelCar container image:
$ podman run --rm -it \ --device nvidia.com/gpu=all \ --security-opt=label=disable \ --shm-size=4g \ --userns=keep-id:uid=1001 \ -p 8000:8000 \ -e HF_HUB_OFFLINE=1 \ -e TRANSFORMERS_OFFLINE=1 \ --mount type=image,source=registry.redhat.io/rhelai1/modelcar-granite-8b-code-instruct:1.4,destination=/model \ registry.redhat.io/rhaii-early-access/vllm-cuda-rhel9:3.4.0-ea.2 \ --model /model/models \ --port 8000NoteThe
--device nvidia.com/gpu=alloption is specific to NVIDIA GPUs. If you are using AMD ROCm GPUs, use--device /dev/kfd --device /dev/driinstead and set--tensor-parallel-sizeto match the number of available GPUs. For the complete AMD ROCm deployment procedure, see Serving and inferencing with Podman using AMD ROCm AI accelerators.
2.3. OCI artifact images
OCI artifact images use the OCI artifact specification to distribute model weights as container registry artifacts rather than as container images. OCI artifact images are distinct from ModelCar container images and require different tooling.
OCI artifact images are designed for use with Red Hat OpenShift AI on OpenShift Container Platform, where the model serving infrastructure handles artifact retrieval natively. If you are using Red Hat AI Inference Server with Podman on RHEL AI, use either Hugging Face models or ModelCar container images instead.
Use OCI artifact images in the following scenarios:
- You are deploying models on OpenShift Container Platform with Red Hat OpenShift AI.
- Your deployment platform supports OCI artifact retrieval natively.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}