Chapter 2. Gateway discovery for Distributed Inference with llm-d

Procedure

1. Enable Gateway discovery in the dashboard configuration:

   $ oc patch odhdashboardconfig odh-dashboard-config \
   -n redhat-ods-applications \
   --type merge \
   -p '{"spec":{"dashboardConfig":{"llmGatewayField":true}}}'

Verification

1. Log in to the OpenShift AI dashboard.
2. In the navigation menu, click AI Hub.
3. Click a project name, then click Deploy model.
4. Verify that the Gateway field is displayed in Advanced Settings.

Procedure

1. Log in to the OpenShift AI dashboard.
2. In the navigation menu, click AI Hub.
3. Click the name of the project that you want to deploy a model in.
4. Click Deploy model.
5. Configure the model deployment properties as required for your model. For more information, see Deploying models on the single-model serving platform.
6. In Advanced Settings, locate the Gateway Selection section.
7. In the Gateway section, locate the Select a gateway dropdown.
8. Select the Gateway from the dropdown list.
9. Click Next.
10. Click Deploy Model.

Verification

1. On the Models page, verify that the LLMInferenceService is listed with a checkmark in the Status column.
2. Click the deployed model name to view details.

3. Verify that the Gateway reference is configured in the deployed model resource:

   $ oc get llminferenceservices <llmisvc_name> -n <namespace> -o yaml

   where:

   <llmisvc_name>

   Specifies the name of your LLMInferenceService.

   <namespace>

   Specifies the project namespace where you deployed the model.

   The output includes the Gateway reference configuration. The Gateway reference appears under spec.router.gateway.

Procedure

1. Create a YAML file for your LLMInferenceService with Gateway references:

   apiVersion: serving.kserve.io/v1alpha1
   kind: LLMInferenceService
   metadata:
     name: <llmisvc_name>
     namespace: <namespace>
   spec:
     modelSource:
       storageUri: <storage_uri>
     runtime:
       name: <runtime_name>
     router:
       gateway:
         refs:
           - name: <gateway_name>
             namespace: <gateway_namespace>

   where:

   <llmisvc_name>

   Specifies the name for your LLMInferenceService.

   <namespace>

   Specifies your project namespace.

   <storage_uri>

   Specifies the location of your model, such as s3://my-bucket/my-model/ or pvc://my-pvc/my-model/.

   <runtime_name>

   Specifies the serving runtime to use, such as vllm-runtime.

   <gateway_name>

   Specifies the name of the Gateway to use.

   <gateway_namespace>

   Specifies the namespace where the Gateway is located.

2. To reference multiple Gateways, add additional entries to the refs list:

     router:
       gateway:
         refs:
           - name: <gateway_name_1>
             namespace: <gateway_namespace_1>
           - name: <gateway_name_2>
             namespace: <gateway_namespace_2>

3. Apply the YAML file:

   $ oc apply -f <filename>.yaml

Verification

1. Verify that the LLMInferenceService was created:

   $ oc get llminferenceservices <llmisvc_name> -n <namespace>

   where:

   <llmisvc_name>

   Specifies the name of your LLMInferenceService.

   <namespace>

   Specifies your project namespace.

2. Verify the Gateway references in the deployed resource:

   $ oc get llminferenceservices <llmisvc_name> -n <namespace> -o yaml

   The output includes the Gateway reference configuration under spec.router.gateway.refs.

3. Verify that the LLMInferenceService is ready:

   $ oc get llminferenceservices <llmisvc_name> -n <namespace> -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

   The output should be True when the service is ready.