SupportConsoleDevelopersStart a trialContact

Chapter 3. Distributed Inference with llm-d core components

Distributed Inference with llm-d deploys a distributed inference stack through Helm charts packaged as OCI container images, integrating industry-standard Kubernetes components to provide intelligent model serving, secure communication, and scalable resource management.

The Distributed Inference with llm-d inference stack consists of the following core components:

Red Hat AI Inference
High-throughput, low-latency LLM inference engine that runs in worker pods and processes model inference requests.
KServe LLMISvc Controller
Kubernetes-native model serving control plane that defines the LLMInferenceService custom resource and manages model lifecycle, deployment orchestration, and operational governance. KServe reconciles the required serving state declared in the custom resource, coordinates component deployment, and exposes OpenAI-compatible APIs. Request routing decisions are handled by the inference scheduler, also known as the Endpoint Picker (EPP).
Inference scheduler (Endpoint Picker)
Scheduling service that makes per-request routing decisions by integrating with the gateway through the ext-proc protocol. The scheduler uses a pluggable system of filters, scorers, and profile handlers to evaluate KV cache locality, queue depth, session affinity, and load distribution when selecting optimal model-serving pods.
Routing sidecar
Sidecar service deployed alongside each model-serving pod that orchestrates disaggregated inference execution. The routing sidecar coordinates prefill and decode stages, manages KV cache transfers between pods, and provides endpoint protection.
Gateway API with Red Hat Connectivity Link
Kubernetes standard API for traffic routing and load balancing, with the Gateway API Inference Extension specification for intelligent request routing. Red Hat Connectivity Link provides the Gateway API implementation.
Istio/Sail Operator
Service mesh that provides mutual TLS (mTLS) encryption, traffic routing, and observability across the inference stack.
cert-manager
Automates Transport Layer Security (TLS) certificate management and provides a Certificate Authority for mutual TLS (mTLS) communication between inference components.
LeaderWorkerSet

Kubernetes workload API for coordinating distributed inference servers and managing pod topology for prefill-decode disaggregation and expert parallelism.

Note

The LeaderWorkerSet Operator is only required for wide expert parallelism (wideEP) with Distributed Inference with llm-d. With LeaderWorkerSet, you can do multi-node deployments across multiple AI accelerator nodes.

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top