Chapter 15. Launch a WideEP inference service with DP-aware routing

Procedure

1. Determine the parallelism configuration for your model.

   Calculate the values for the .spec.parallelism fields based on your cluster topology:

   • data: The total number of DP ranks across the cluster, calculated as (nodes * GPUs_per_node) / tensor. For example, 4 nodes with 8 GPUs each and tensor: 1 gives data: 32.
   • dataLocal: The number of DP ranks per node, which must match the number of GPUs per node. For example, dataLocal: 8 for nodes with 8 GPUs.
   • expert: Set to true to enable expert parallelism.
   • tensor: Set to 1 when using expert parallelism as the primary distribution strategy.

   The formula data / dataLocal determines the number of pods. For example, data: 32 with dataLocal: 8 produces 4 pods.

2. Create an LLMInferenceService CR that uses external multi-port DP mode.

   Create a file called wideep-inference-service.yaml with the following content:

   apiVersion: serving.kserve.io/v1alpha2
   kind: LLMInferenceService
   metadata:
     name: deepseek-r1-wideep
     namespace: llm-inference
   spec:
     model:
       uri: pvc://model-storage
       name: deepseek-ai/DeepSeek-R1-0528
     replicas: 1
     parallelism:
       data: 32
       dataLocal: 8
       expert: true
       tensor: 1
     router: # Uses default routing settings
       scheduler: {}
       route: {}
       gateway: {}
     template:
       containers:
       - name: main
         args:
           - --data-parallel-multi-port-external-lb
           - --data-parallel-size=32
           - --data-parallel-size-local=8
           - --data-parallel-supervisor-port=9256
           - --port=8000
           - --all2all-backend=deepep_high_throughput
         resources:
           limits:
             cpu: "32"
             memory: 256Gi
             nvidia.com/gpu: "8"
           requests:
             cpu: "16"
             memory: 128Gi
             nvidia.com/gpu: "8"
         startupProbe:
           httpGet:
             path: /readyz
             port: 9256
             scheme: HTTP
           periodSeconds: 10
           failureThreshold: 60
           timeoutSeconds: 5
         readinessProbe:
           httpGet:
             path: /readyz
             port: 9256
             scheme: HTTP
           periodSeconds: 5
           failureThreshold: 3
           timeoutSeconds: 5
         livenessProbe:
           httpGet:
             path: /health
             port: 9256
             scheme: HTTP
           periodSeconds: 15
           failureThreshold: 5
           timeoutSeconds: 5
       terminationGracePeriodSeconds: 120

   where:

   • --data-parallel-multi-port-external-lb - Enables external multi-port DP mode. Each local DP rank gets its own serving port so the Endpoint Picker can route to individual ranks. The DP Supervisor starts the local rank processes and aggregates their health on the admin port.
   • --data-parallel-size=32 - Specifies the total number of DP ranks across the cluster. Must match the spec.parallelism.data value.
   • --data-parallel-size-local=8 - Specifies the number of DP ranks per pod. Must match the spec.parallelism.dataLocal value and the number of GPUs per node.
   • --data-parallel-supervisor-port=9256 - Specifies the admin port for aggregated /health and /readyz. Point all Kubernetes probes at this port, not at per-rank serving ports.
   • --port=8000 - Specifies the base serving port for inference traffic. Each DP rank listens on a sequential port: 8000, 8001, 8002, through 8007 for 8 local ranks. The Endpoint Picker targets these ports when scheduling requests.

   • --all2all-backend=deepep_high_throughput - Specifies the all-to-all communication backend for expert parallelism. The deepep_high_throughput backend is optimized for batch processing throughput and requires RDMA-capable networking such as RoCE or InfiniBand. For disaggregated deployments that separate prefill and decode stages, use deepep_low_latency for the decode stage.

     port: 9256 in probe configurations

     Targets the DP Supervisor admin port for all probes. Do not target per-rank serving ports.

   • failureThreshold: 60 in startupProbe - Allows up to 600 seconds for model loading. Calculate this value as ceiling(max_startup_seconds / periodSeconds).
   • terminationGracePeriodSeconds: 120 - Allows the DP Supervisor to forward SIGTERM to all child processes and wait for graceful drain.

3. Deploy the inference service:

   $ oc apply -f wideep-inference-service.yaml

4. Monitor the startup progress.

   Large MoE models can take 7-10 minutes or more to load. Check the pod status during startup:

   $ oc get pods -n llm-inference -w

   The pod transitions through the following states:

   • Pending: Waiting for GPU node scheduling.
   • ContainerCreating: Pulling the container image.
   • Running with startup probe active: Model is loading. DP ranks initialize and the supervisor waits for all DP ranks to become healthy.
   • Running with all probes passing: All DP ranks are ready and the supervisor is serving health on the admin port.

5. Verify that terminationGracePeriodSeconds is set for coordinated shutdown.

   The DP Supervisor requires enough time to forward SIGTERM to all child DP rank processes and wait for graceful drain. The recommended range is 60-120 seconds. The supervisor waits for the configured shutdown_timeout plus 5 seconds before force-killing remaining processes.

   Note

   If you created the LLMInferenceService CR without terminationGracePeriodSeconds, patch the CR to add the specification:

   $ oc patch llmisvc deepseek-r1-wideep -n llm-inference --type=merge \
     -p '{"spec":{"template":{"terminationGracePeriodSeconds":120}}}'