Chapter 5. Installing the single-model serving platform

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   Copy to Clipboard Toggle word wrap

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

2. Create the required namespace for Red Hat OpenShift Service Mesh.

   Copy to Clipboard Toggle word wrap

   $ oc create ns istio-system

   You see the following output:

   Copy to Clipboard Toggle word wrap

   namespace/istio-system created

3. Define a ServiceMeshControlPlane object in a YAML file named smcp.yaml with the following contents:

   Copy to Clipboard Toggle word wrap

   apiVersion: maistra.io/v2
   kind: ServiceMeshControlPlane
   metadata:
     name: minimal
     namespace: istio-system
   spec:
     tracing:
       type: None
     addons:
       grafana:
         enabled: false
       kiali:
         name: kiali
         enabled: false
       prometheus:
         enabled: false
       jaeger:
         name: jaeger
     security:
       dataPlane:
         mtls: true
       identity:
         type: ThirdParty
     techPreview:
       meshConfig:
         defaultConfig:
           terminationDrainDuration: 35s
     gateways:
       ingress:
         service:
           metadata:
             labels:
               knative: ingressgateway
     proxy:
       networking:
         trafficControl:
           inbound:
             excludedPorts:
               - 8444
               - 8022

   For more information about the values in the YAML file, see the Service Mesh control plane configuration reference.

4. Create the service mesh control plane.

   Copy to Clipboard Toggle word wrap

   $ oc apply -f smcp.yaml

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   Copy to Clipboard Toggle word wrap

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

2. Check whether the required project (that is, namespace) for Knative Serving already exists.

   Copy to Clipboard Toggle word wrap

   $ oc get ns knative-serving

   If the project exists, you see output similar to the following example:

   Copy to Clipboard Toggle word wrap

   NAME              STATUS   AGE
   knative-serving   Active   4d20h

3. If the knative-serving project doesn’t already exist, create it.

   Copy to Clipboard Toggle word wrap

   $ oc create ns knative-serving

   You see the following output:

   Copy to Clipboard Toggle word wrap

   namespace/knative-serving created

4. Define a ServiceMeshMember object in a YAML file called default-smm.yaml with the following contents:

   Copy to Clipboard Toggle word wrap

   apiVersion: maistra.io/v1
   kind: ServiceMeshMember
   metadata:
     name: default
     namespace: knative-serving
   spec:
     controlPlaneRef:
       namespace: istio-system
       name: minimal

5. Create the ServiceMeshMember object in the istio-system namespace.

   Copy to Clipboard Toggle word wrap

   $ oc apply -f default-smm.yaml

   You see the following output:

   Copy to Clipboard Toggle word wrap

   servicemeshmember.maistra.io/default created

6. Define a KnativeServing object in a YAML file called knativeserving-istio.yaml with the following contents:

   Copy to Clipboard Toggle word wrap

   apiVersion: operator.knative.dev/v1beta1
   kind: KnativeServing
   metadata:
     name: knative-serving
     namespace: knative-serving
     annotations:
       serverless.openshift.io/default-enable-http2: "true"
   spec:
     workloads:
       - name: net-istio-controller
         env:
           - container: controller
             envVars:
               - name: ENABLE_SECRET_INFORMER_FILTERING_BY_CERT_UID
                 value: 'true'
       - annotations:
           sidecar.istio.io/inject: "true" 

   1

   
           sidecar.istio.io/rewriteAppHTTPProbers: "true" 

   2

   
         name: activator
       - annotations:
           sidecar.istio.io/inject: "true"
           sidecar.istio.io/rewriteAppHTTPProbers: "true"
         name: autoscaler
     ingress:
       istio:
         enabled: true
     config:
       features:
         kubernetes.podspec-affinity: enabled
         kubernetes.podspec-nodeselector: enabled
         kubernetes.podspec-tolerations: enabled

   The preceding file defines a custom resource (CR) for a KnativeServing object. The CR also adds the following actions to each of the activator and autoscaler pods:

   1

   Injects an Istio sidecar to the pod. This makes the pod part of the service mesh.

   2

   Enables the Istio sidecar to rewrite the HTTP liveness and readiness probes for the pod.

   Note

   If you configure a custom domain for a Knative service, you can use a TLS certificate to secure the mapped service. To do this, you must create a TLS secret, and then update the DomainMapping CR to use the TLS secret that you have created. For more information, see Securing a mapped service using a TLS certificate in the Red Hat OpenShift Serverless documentation.

7. Create the KnativeServing object in the specified knative-serving namespace.

   Copy to Clipboard Toggle word wrap

   $ oc apply -f knativeserving-istio.yaml

   You see the following output:

   Copy to Clipboard Toggle word wrap

   knativeserving.operator.knative.dev/knative-serving created

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   Copy to Clipboard Toggle word wrap

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Important

   If you have your own wildcard certificate and key to specify when configuring the gateways, skip to step 11 of this procedure.

2. Set environment variables to define base directories for generation of a wildcard certificate and key for the gateways.

   Copy to Clipboard Toggle word wrap

   $ export BASE_DIR=/tmp/kserve
   $ export BASE_CERT_DIR=${BASE_DIR}/certs

3. Set an environment variable to define the common name used by the ingress controller of your OpenShift cluster.

   Copy to Clipboard Toggle word wrap

   $ export COMMON_NAME=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}' | awk -F'.' '{print $(NF-1)"."$NF}')

4. Set an environment variable to define the domain name used by the ingress controller of your OpenShift cluster.

   Copy to Clipboard Toggle word wrap

   $ export DOMAIN_NAME=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}')

5. Create the required base directories for the certificate generation, based on the environment variables that you previously set.

   Copy to Clipboard Toggle word wrap

   $ mkdir ${BASE_DIR}
   $ mkdir ${BASE_CERT_DIR}

6. Create the OpenSSL configuration for generation of a wildcard certificate.

   Copy to Clipboard Toggle word wrap

   $ cat <<EOF> ${BASE_DIR}/openssl-san.config
   [ req ]
   distinguished_name = req
   [ san ]
   subjectAltName = DNS:*.${DOMAIN_NAME}
   EOF

7. Generate a root certificate.

   Copy to Clipboard Toggle word wrap

   $ openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 \
   -subj "/O=Example Inc./CN=${COMMON_NAME}" \
   -keyout ${BASE_CERT_DIR}/root.key \
   -out ${BASE_CERT_DIR}/root.crt

8. Generate a wildcard certificate signed by the root certificate.

   Copy to Clipboard Toggle word wrap

   $ openssl req -x509 -newkey rsa:2048 \
   -sha256 -days 3560 -nodes \
   -subj "/CN=${COMMON_NAME}/O=Example Inc." \
   -extensions san -config ${BASE_DIR}/openssl-san.config \
   -CA ${BASE_CERT_DIR}/root.crt \
   -CAkey ${BASE_CERT_DIR}/root.key \
   -keyout ${BASE_CERT_DIR}/wildcard.key  \
   -out ${BASE_CERT_DIR}/wildcard.crt
   
   $ openssl x509 -in ${BASE_CERT_DIR}/wildcard.crt -text

9. Verify the wildcard certificate.

   Copy to Clipboard Toggle word wrap

   $ openssl verify -CAfile ${BASE_CERT_DIR}/root.crt ${BASE_CERT_DIR}/wildcard.crt

10. Export the wildcard key and certificate that were created by the script to new environment variables.

    Copy to Clipboard Toggle word wrap

    $ export TARGET_CUSTOM_CERT=${BASE_CERT_DIR}/wildcard.crt
    $ export TARGET_CUSTOM_KEY=${BASE_CERT_DIR}/wildcard.key

11. Optional: To export your own wildcard key and certificate to new environment variables, enter the following commands:

    Copy to Clipboard Toggle word wrap

    $ export TARGET_CUSTOM_CERT=<path_to_certificate>
    $ export TARGET_CUSTOM_KEY=<path_to_key>

    Note

    In the certificate that you provide, you must specify the domain name used by the ingress controller of your OpenShift cluster. You can check this value by running the following command:

    $ oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}'

12. Create a TLS secret in the istio-system namespace using the environment variables that you set for the wildcard certificate and key.

    Copy to Clipboard Toggle word wrap

    $ oc create secret tls wildcard-certs --cert=${TARGET_CUSTOM_CERT} --key=${TARGET_CUSTOM_KEY} -n istio-system

13. Create a gateways.yaml YAML file with the following contents:

    Copy to Clipboard Toggle word wrap

    apiVersion: v1
    kind: Service 

    1

    
    metadata:
      labels:
        experimental.istio.io/disable-gateway-port-translation: "true"
      name: knative-local-gateway
      namespace: istio-system
    spec:
      ports:
        - name: http2
          port: 80
          protocol: TCP
          targetPort: 8081
      selector:
        knative: ingressgateway
      type: ClusterIP
    ---
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
      name: knative-ingress-gateway 

    2

    
      namespace: knative-serving
    spec:
      selector:
        knative: ingressgateway
      servers:
        - hosts:
            - '*'
          port:
            name: https
            number: 443
            protocol: HTTPS
          tls:
            credentialName: wildcard-certs
            mode: SIMPLE
    ---
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
     name: knative-local-gateway 

    3

    
     namespace: knative-serving
    spec:
     selector:
       knative: ingressgateway
     servers:
       - port:
           number: 8081
           name: https
           protocol: HTTPS
         tls:
           mode: ISTIO_MUTUAL
         hosts:
           - "*"

    1

    Defines a service in the istio-system namespace for the Knative local gateway.

    2

    Defines an ingress gateway in the knative-serving namespace. The gateway uses the TLS secret you created earlier in this procedure. The ingress gateway handles external traffic to Knative.

    3

    Defines a local gateway for Knative in the knative-serving namespace.

14. Apply the gateways.yaml file to create the defined resources.

    Copy to Clipboard Toggle word wrap

    $ oc apply -f gateways.yaml

    You see the following output:

    Copy to Clipboard Toggle word wrap

    service/knative-local-gateway created
    gateway.networking.istio.io/knative-ingress-gateway created
    gateway.networking.istio.io/knative-local-gateway created

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator.

3. For installation of KServe, configure the OpenShift Service Mesh component as follows:

   1. Click the DSC Initialization tab.
   2. Click the default-dsci object.
   3. Click the YAML tab.

   4. In the spec section, add and configure the serviceMesh component as shown:

      Copy to Clipboard Toggle word wrap

      spec:
       serviceMesh:
         managementState: Unmanaged

   5. Click Save.

4. For installation of KServe, configure the KServe and OpenShift Serverless components as follows:

   1. In the web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator.
   2. Click the Data Science Cluster tab.
   3. Click the default-dsc DSC object.
   4. Click the YAML tab.

   5. In the spec.components section, configure the kserve component as shown:

      Copy to Clipboard Toggle word wrap

      spec:
       components:
         kserve:
           managementState: Managed

   6. Within the kserve component, add the serving component, and configure it as shown:

      Copy to Clipboard Toggle word wrap

      spec:
       components:
         kserve:
           managementState: Managed
           serving:
             managementState: Unmanaged

   7. Click Save.

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator.

3. Disable the OpenShift Service Mesh component as follows:

   1. Click the DSC Initialization tab.
   2. Click the default-dsci object.
   3. Click the YAML tab.

   4. In the spec section, add the serviceMesh component (if it is not already present) and configure the managementState field as shown:

      Copy to Clipboard Toggle word wrap

      spec:
       serviceMesh:
         managementState: Removed

   5. Click Save.

Verification

1. In the web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator.

   The Operator details page opens.

2. In the Conditions section, confirm that there is no ReconcileComplete condition with a status value of Unknown.

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators OperatorHub.
3. On the OperatorHub page, in the Filter by keyword field, type Red Hat - Authorino.
4. Click the Red Hat - Authorino Operator.
5. On the Red Hat - Authorino Operator page, review the Operator information and then click Install.
6. On the Install Operator page, keep the default values for Update channel, Version, Installation mode, Installed Namespace and Update Approval.
7. Click Install.

Procedure

1. Open a new terminal window.

2. Log in to the OpenShift command-line interface (CLI) as follows:

   Copy to Clipboard Toggle word wrap

   $ oc login <openshift_cluster_url> -u <username> -p <password>

3. Create a namespace to install the Authorino instance.

   Copy to Clipboard Toggle word wrap

   $ oc new-project <namespace_for_authorino_instance>

   Note

   The automated installation process creates a namespace called redhat-ods-applications-auth-provider for the Authorino instance. Consider using the same namespace name for the manual installation.

4. To enroll the new namespace for the Authorino instance in your existing OpenShift Service Mesh instance, create a new YAML file with the following contents:

   Copy to Clipboard Toggle word wrap

   apiVersion: maistra.io/v1
   kind: ServiceMeshMember
   metadata:
     name: default
     namespace: <namespace_for_authorino_instance>
   spec:
     controlPlaneRef:
       namespace: <namespace_for_service_mesh_instance>
       name: <name_of_service_mesh_instance>

5. Save the YAML file.

6. Create the ServiceMeshMember resource on your cluster.

   Copy to Clipboard Toggle word wrap

   $ oc create -f <file_name>.yaml

7. To configure an Authorino instance, create a new YAML file as shown in the following example:

   Copy to Clipboard Toggle word wrap

   apiVersion: operator.authorino.kuadrant.io/v1beta1
   kind: Authorino
   metadata:
     name: authorino
     namespace: <namespace_for_authorino_instance>
   spec:
     authConfigLabelSelectors: security.opendatahub.io/authorization-group=default
     clusterWide: true
     listener:
       tls:
         enabled: false
     oidcServer:
       tls:
         enabled: false

8. Save the YAML file.

9. Create the Authorino resource on your cluster.

   Copy to Clipboard Toggle word wrap

   $ oc create -f <file_name>.yaml

10. Patch the Authorino deployment to inject an Istio sidecar, which makes the Authorino instance part of your OpenShift Service Mesh instance.

    Copy to Clipboard Toggle word wrap

    $ oc patch deployment <name_of_authorino_instance> -n <namespace_for_authorino_instance> -p '{"spec": {"template":{"metadata":{"labels":{"sidecar.istio.io/inject":"true"}}}} }'

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a user that has privileges to update the OpenShift Service Mesh instance, log in to the OpenShift CLI as shown in the following example:

   Copy to Clipboard Toggle word wrap

   $ oc login <openshift_cluster_url> -u <username> -p <password>

2. Create a new YAML file with the following contents:

   Copy to Clipboard Toggle word wrap

   spec:
    techPreview:
      meshConfig:
        extensionProviders:
        - name: redhat-ods-applications-auth-provider
          envoyExtAuthzGrpc:
            service: <name_of_authorino_instance>-authorino-authorization.<namespace_for_authorino_instance>.svc.cluster.local
            port: 50051

3. Save the YAML file.

4. Use the oc patch command to apply the YAML file to your OpenShift Service Mesh instance.

   Copy to Clipboard Toggle word wrap

   $ oc patch smcp <name_of_service_mesh_instance> --type merge -n <namespace_for_service_mesh_instance> --patch-file <file_name>.yaml

   Important

   You can apply the configuration shown as a patch only if you have not already specified other extension providers in your OpenShift Service Mesh instance. If you have already specified other extension providers, you must manually edit your ServiceMeshControlPlane resource to add the configuration.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a user that has privileges to update the KServe deployment, log in to the OpenShift CLI as shown in the following example:

   Copy to Clipboard Toggle word wrap

   $ oc login <openshift_cluster_url> -u <username> -p <password>

2. Create a new YAML file with the following contents:

   Copy to Clipboard Toggle word wrap

   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: kserve-predictor
   spec:
     action: CUSTOM
     provider:
        name: redhat-ods-applications-auth-provider 

   1

   
     rules:
        - to:
             - operation:
                  notPaths:
                     - /healthz
                     - /debug/pprof/
                     - /metrics
                     - /wait-for-drain
     selector:
        matchLabels:
           component: predictor

   1

   The name that you specify must match the name of the extension provider that you added to your OpenShift Service Mesh instance.

3. Save the YAML file.

4. Create the AuthorizationPolicy resource in the namespace for your OpenShift Service Mesh instance.

   Copy to Clipboard Toggle word wrap

   $ oc create -n <namespace_for_service_mesh_instance> -f <file_name>.yaml

5. Create another new YAML file with the following contents:

   Copy to Clipboard Toggle word wrap

   apiVersion: networking.istio.io/v1alpha3
   kind: EnvoyFilter
   metadata:
     name: activator-host-header
   spec:
     priority: 20
     workloadSelector:
       labels:
         component: predictor
     configPatches:
     - applyTo: HTTP_FILTER
       match:
         listener:
           filterChain:
             filter:
               name: envoy.filters.network.http_connection_manager
       patch:
         operation: INSERT_BEFORE
         value:
           name: envoy.filters.http.lua
           typed_config:
             '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
             inlineCode: |
              function envoy_on_request(request_handle)
                 local headers = request_handle:headers()
                 if not headers then
                   return
                 end
                 local original_host = headers:get("k-original-host")
                 if original_host then
                   port_seperator = string.find(original_host, ":", 7)
                   if port_seperator then
                     original_host = string.sub(original_host, 0, port_seperator-1)
                   end
                   headers:replace('host', original_host)
                 end
               end

   The EnvoyFilter resource shown continually resets the HTTP host header to the one initially included in any inference request.

6. Create the EnvoyFilter resource in the namespace for your OpenShift Service Mesh instance.

   Copy to Clipboard Toggle word wrap

   $ oc create -n <namespace_for_service_mesh_instance> -f <file_name>.yaml