Chapter 4. Distributed inference workloads

Distributed Inference with llm-d uses a layered architecture to distribute inference workloads across a Kubernetes cluster.

Client requests flow through an inference gateway that routes the requests to optimized model replicas, with intelligent scheduling and autoscaling to manage resources efficiently. The gateway provides OpenAI-compatible APIs with streaming token responses. Body-based routing selects the InferencePool from the model name specified in the API request.

The inference pool contains model-serving variants, such as Prefill and Decode, each running Red Hat AI Inference instances. Variants handle different phases of the inference process. Shared prefix caching enables key-value (KV) cache sharing between model-serving variants.

Figure 4.1. Distributed Inference with llm-d processing architecture

Body-based routing selects the InferencePool from the model name in the OpenAI-compatible API spec.

The inference scheduler is an extensible framework of filters, scorers, and profile handlers that provides load, KV, and prefill/decode-aware routing. The scheduler evaluates GPU utilization, queue depth, cache residency, and service level agreement (SLA) constraints to direct each request to the optimal replica. Routing favors replicas with warm KV cache entries for the request’s prefix, maximizing cache reuse and avoiding redundant computation.

Each decode pod asynchronously reports KV cache availability and load metrics to the inference scheduler.