Chapter 5. Installing the single-model serving platform

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy to Clipboard

2. Create the required namespace for Red Hat OpenShift Service Mesh.

   $ oc create ns istio-system

   Copy to Clipboard

   You see the following output:

   namespace/istio-system created

   Copy to Clipboard

3. Define a ServiceMeshControlPlane object in a YAML file named smcp.yaml with the following contents:

   apiVersion: maistra.io/v2
   kind: ServiceMeshControlPlane
   metadata:
     name: minimal
     namespace: istio-system
   spec:
     tracing:
       type: None
     addons:
       grafana:
         enabled: false
       kiali:
         name: kiali
         enabled: false
       prometheus:
         enabled: false
       jaeger:
         name: jaeger
     security:
       dataPlane:
         mtls: true
       identity:
         type: ThirdParty
     techPreview:
       meshConfig:
         defaultConfig:
           terminationDrainDuration: 35s
     gateways:
       ingress:
         service:
           metadata:
             labels:
               knative: ingressgateway
     proxy:
       networking:
         trafficControl:
           inbound:
             excludedPorts:
               - 8444
               - 8022

   Copy to Clipboard

   For more information about the values in the YAML file, see the Service Mesh control plane configuration reference.

4. Create the service mesh control plane.

   $ oc apply -f smcp.yaml

   Copy to Clipboard

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy to Clipboard

2. Check whether the required project (that is, namespace) for Knative Serving already exists.

   $ oc get ns knative-serving

   Copy to Clipboard

   If the project exists, you see output similar to the following example:

   NAME              STATUS   AGE
   knative-serving   Active   4d20h

   Copy to Clipboard

3. If the knative-serving project doesn’t already exist, create it.

   $ oc create ns knative-serving

   Copy to Clipboard

   You see the following output:

   namespace/knative-serving created

   Copy to Clipboard

4. Define a ServiceMeshMember object in a YAML file called default-smm.yaml with the following contents:

   apiVersion: maistra.io/v1
   kind: ServiceMeshMember
   metadata:
     name: default
     namespace: knative-serving
   spec:
     controlPlaneRef:
       namespace: istio-system
       name: minimal

   Copy to Clipboard

5. Create the ServiceMeshMember object in the istio-system namespace.

   $ oc apply -f default-smm.yaml

   Copy to Clipboard

   You see the following output:

   servicemeshmember.maistra.io/default created

   Copy to Clipboard

6. Define a KnativeServing object in a YAML file called knativeserving-istio.yaml with the following contents:

   apiVersion: operator.knative.dev/v1beta1
   kind: KnativeServing
   metadata:
     name: knative-serving
     namespace: knative-serving
     annotations:
       serverless.openshift.io/default-enable-http2: "true"
   spec:
     workloads:
       - name: net-istio-controller
         env:
           - container: controller
             envVars:
               - name: ENABLE_SECRET_INFORMER_FILTERING_BY_CERT_UID
                 value: 'true'
       - annotations:
           sidecar.istio.io/inject: "true" 

   1

   
           sidecar.istio.io/rewriteAppHTTPProbers: "true" 

   2

   
         name: activator
       - annotations:
           sidecar.istio.io/inject: "true"
           sidecar.istio.io/rewriteAppHTTPProbers: "true"
         name: autoscaler
     ingress:
       istio:
         enabled: true
     config:
       features:
         kubernetes.podspec-affinity: enabled
         kubernetes.podspec-nodeselector: enabled
         kubernetes.podspec-tolerations: enabled

   Copy to Clipboard

   The preceding file defines a custom resource (CR) for a KnativeServing object. The CR also adds the following actions to each of the activator and autoscaler pods:

   1

   Injects an Istio sidecar to the pod. This makes the pod part of the service mesh.

   2

   Enables the Istio sidecar to rewrite the HTTP liveness and readiness probes for the pod.

   Note

   If you configure a custom domain for a Knative service, you can use a TLS certificate to secure the mapped service. To do this, you must create a TLS secret, and then update the DomainMapping CR to use the TLS secret that you have created. For more information, see Securing a mapped service using a TLS certificate in the Red Hat OpenShift Serverless documentation.

7. Optional: You can configure knative serving to allow zero initial scale for revisions by using the allow-zero-initial-scale autoscaler configuration:

   spec:
     config:
       autoscaler:
         allow-zero-initial-scale: "true"

   Copy to Clipboard

8. Optional: You can enable initContainers for your knative serving instance by adding the kubernetes.podspec-init-containers feature flag as shown in the following example:

   spec:
     config:
       features:
         kubernetes.podspec-init-containers: "enabled"

   Copy to Clipboard

   You can also add the flag in the config-feature ConfigMap in the knative-serving namespace. For more information, see Feature and extension flags.

9. Create the KnativeServing object in the specified knative-serving namespace.

   $ oc apply -f knativeserving-istio.yaml

   Copy to Clipboard

   You see the following output:

   knativeserving.operator.knative.dev/knative-serving created

   Copy to Clipboard

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy to Clipboard
   Important

   If you have your own wildcard certificate and key to specify when configuring the gateways, skip to step 11 of this procedure.

2. Set environment variables to define base directories for generation of a wildcard certificate and key for the gateways.

   $ export BASE_DIR=/tmp/kserve
   $ export BASE_CERT_DIR=${BASE_DIR}/certs

   Copy to Clipboard

3. Set an environment variable to define the common name used by the ingress controller of your OpenShift cluster.

   $ export COMMON_NAME=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}' | awk -F'.' '{print $(NF-1)"."$NF}')

   Copy to Clipboard

4. Set an environment variable to define the domain name used by the ingress controller of your OpenShift cluster.

   $ export DOMAIN_NAME=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}')

   Copy to Clipboard

5. Create the required base directories for the certificate generation, based on the environment variables that you previously set.

   $ mkdir ${BASE_DIR}
   $ mkdir ${BASE_CERT_DIR}

   Copy to Clipboard

6. Create the OpenSSL configuration for generation of a wildcard certificate.

   $ cat <<EOF> ${BASE_DIR}/openssl-san.config
   [ req ]
   distinguished_name = req
   [ san ]
   subjectAltName = DNS:*.${DOMAIN_NAME}
   EOF

   Copy to Clipboard

7. Generate a root certificate.

   $ openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 \
   -subj "/O=Example Inc./CN=${COMMON_NAME}" \
   -keyout ${BASE_CERT_DIR}/root.key \
   -out ${BASE_CERT_DIR}/root.crt

   Copy to Clipboard

8. Generate a wildcard certificate signed by the root certificate.

   $ openssl req -x509 -newkey rsa:2048 \
   -sha256 -days 3560 -nodes \
   -subj "/CN=${COMMON_NAME}/O=Example Inc." \
   -extensions san -config ${BASE_DIR}/openssl-san.config \
   -CA ${BASE_CERT_DIR}/root.crt \
   -CAkey ${BASE_CERT_DIR}/root.key \
   -keyout ${BASE_CERT_DIR}/wildcard.key  \
   -out ${BASE_CERT_DIR}/wildcard.crt
   
   $ openssl x509 -in ${BASE_CERT_DIR}/wildcard.crt -text

   Copy to Clipboard

9. Verify the wildcard certificate.

   $ openssl verify -CAfile ${BASE_CERT_DIR}/root.crt ${BASE_CERT_DIR}/wildcard.crt

   Copy to Clipboard

10. Export the wildcard key and certificate that were created by the script to new environment variables.

    $ export TARGET_CUSTOM_CERT=${BASE_CERT_DIR}/wildcard.crt
    $ export TARGET_CUSTOM_KEY=${BASE_CERT_DIR}/wildcard.key

    Copy to Clipboard

11. Optional: To export your own wildcard key and certificate to new environment variables, enter the following commands:

    $ export TARGET_CUSTOM_CERT=<path_to_certificate>
    $ export TARGET_CUSTOM_KEY=<path_to_key>

    Copy to Clipboard
    Note

    In the certificate that you provide, you must specify the domain name used by the ingress controller of your OpenShift cluster. You can check this value by running the following command:

    $ oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}'

12. Create a TLS secret in the istio-system namespace using the environment variables that you set for the wildcard certificate and key.

    $ oc create secret tls wildcard-certs --cert=${TARGET_CUSTOM_CERT} --key=${TARGET_CUSTOM_KEY} -n istio-system

    Copy to Clipboard

13. Create a gateways.yaml YAML file with the following contents:

    apiVersion: v1
    kind: Service 

    1

    
    metadata:
      labels:
        experimental.istio.io/disable-gateway-port-translation: "true"
      name: knative-local-gateway
      namespace: istio-system
    spec:
      ports:
        - name: http2
          port: 80
          protocol: TCP
          targetPort: 8081
      selector:
        knative: ingressgateway
      type: ClusterIP
    ---
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
      name: knative-ingress-gateway 

    2

    
      namespace: knative-serving
    spec:
      selector:
        knative: ingressgateway
      servers:
        - hosts:
            - '*'
          port:
            name: https
            number: 443
            protocol: HTTPS
          tls:
            credentialName: wildcard-certs
            mode: SIMPLE
    ---
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
     name: knative-local-gateway 

    3

    
     namespace: knative-serving
    spec:
     selector:
       knative: ingressgateway
     servers:
       - port:
           number: 8081
           name: https
           protocol: HTTPS
         tls:
           mode: ISTIO_MUTUAL
         hosts:
           - "*"

    Copy to Clipboard

    1

    Defines a service in the istio-system namespace for the Knative local gateway.

    2

    Defines an ingress gateway in the knative-serving namespace. The gateway uses the TLS secret you created earlier in this procedure. The ingress gateway handles external traffic to Knative.

    3

    Defines a local gateway for Knative in the knative-serving namespace.

14. Apply the gateways.yaml file to create the defined resources.

    $ oc apply -f gateways.yaml

    Copy to Clipboard

    You see the following output:

    service/knative-local-gateway created
    gateway.networking.istio.io/knative-ingress-gateway created
    gateway.networking.istio.io/knative-local-gateway created

    Copy to Clipboard

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator.

3. For installation of KServe, configure the OpenShift Service Mesh component as follows:

   1. Click the DSC Initialization tab.
   2. Click the default-dsci object.
   3. Click the YAML tab.

   4. In the spec section, add and configure the serviceMesh component as shown:

      spec:
       serviceMesh:
         managementState: Unmanaged

      Copy to Clipboard
   5. Click Save.

4. For installation of KServe, configure the KServe and OpenShift Serverless components as follows:

   1. In the web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator.
   2. Click the Data Science Cluster tab.
   3. Click the default-dsc DSC object.
   4. Click the YAML tab.

   5. In the spec.components section, configure the kserve component as shown:

      spec:
       components:
         kserve:
           managementState: Managed

      Copy to Clipboard

   6. Within the kserve component, add the serving component, and configure it as shown:

      spec:
       components:
         kserve:
           managementState: Managed
           serving:
             managementState: Unmanaged

      Copy to Clipboard
   7. Click Save.

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator.

3. Disable the OpenShift Service Mesh component as follows:

   1. Click the DSC Initialization tab.
   2. Click the default-dsci object.
   3. Click the YAML tab.

   4. In the spec section, add the serviceMesh component (if it is not already present) and configure the managementState field as shown:

      spec:
       serviceMesh:
         managementState: Removed

      Copy to Clipboard
   5. Click Save.

Verification

1. In the web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator.

   The Operator details page opens.

2. In the Conditions section, confirm that there is no ReconcileComplete condition with a status value of Unknown.

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators OperatorHub.
3. On the OperatorHub page, in the Filter by keyword field, type Red Hat - Authorino.
4. Click the Red Hat - Authorino Operator.
5. On the Red Hat - Authorino Operator page, review the Operator information and then click Install.

6. On the Install Operator page, keep the default values for Installation mode, Installed Namespace and Update Approval.

   1. For Update channel, select Stable.
   2. For Version, select 1.2.1 or later.
7. Click Install.

Procedure

1. Open a new terminal window.

2. Log in to the OpenShift command-line interface (CLI) as follows:

   $ oc login <openshift_cluster_url> -u <username> -p <password>

   Copy to Clipboard

3. Create a namespace to install the Authorino instance.

   $ oc new-project <namespace_for_authorino_instance>

   Copy to Clipboard
   Note

   The automated installation process creates a namespace called redhat-ods-applications-auth-provider for the Authorino instance. Consider using the same namespace name for the manual installation.

4. To enroll the new namespace for the Authorino instance in your existing OpenShift Service Mesh instance, create a new YAML file with the following contents:

   apiVersion: maistra.io/v1
   kind: ServiceMeshMember
   metadata:
     name: default
     namespace: <namespace_for_authorino_instance>
   spec:
     controlPlaneRef:
       namespace: <namespace_for_service_mesh_instance>
       name: <name_of_service_mesh_instance>

   Copy to Clipboard
5. Save the YAML file.

6. Create the ServiceMeshMember resource on your cluster.

   $ oc create -f <file_name>.yaml

   Copy to Clipboard

7. To configure an Authorino instance, create a new YAML file as shown in the following example:

   apiVersion: operator.authorino.kuadrant.io/v1beta1
   kind: Authorino
   metadata:
     name: authorino
     namespace: <namespace_for_authorino_instance>
   spec:
     authConfigLabelSelectors: security.opendatahub.io/authorization-group=default
     clusterWide: true
     listener:
       tls:
         enabled: false
     oidcServer:
       tls:
         enabled: false

   Copy to Clipboard
8. Save the YAML file.

9. Create the Authorino resource on your cluster.

   $ oc create -f <file_name>.yaml

   Copy to Clipboard

10. Patch the Authorino deployment to inject an Istio sidecar, which makes the Authorino instance part of your OpenShift Service Mesh instance.

    $ oc patch deployment <name_of_authorino_instance> -n <namespace_for_authorino_instance> -p '{"spec": {"template":{"metadata":{"labels":{"sidecar.istio.io/inject":"true"}}}} }'

    Copy to Clipboard

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a user that has privileges to update the OpenShift Service Mesh instance, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <username> -p <password>

   Copy to Clipboard

2. Create a new YAML file with the following contents:

   spec:
    techPreview:
      meshConfig:
        extensionProviders:
        - name: redhat-ods-applications-auth-provider
          envoyExtAuthzGrpc:
            service: <name_of_authorino_instance>-authorino-authorization.<namespace_for_authorino_instance>.svc.cluster.local
            port: 50051

   Copy to Clipboard
3. Save the YAML file.

4. Use the oc patch command to apply the YAML file to your OpenShift Service Mesh instance.

   $ oc patch smcp <name_of_service_mesh_instance> --type merge -n <namespace_for_service_mesh_instance> --patch-file <file_name>.yaml

   Copy to Clipboard
   Important

   You can apply the configuration shown as a patch only if you have not already specified other extension providers in your OpenShift Service Mesh instance. If you have already specified other extension providers, you must manually edit your ServiceMeshControlPlane resource to add the configuration.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a user that has privileges to update the KServe deployment, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <username> -p <password>

   Copy to Clipboard

2. Create a new YAML file with the following contents:

   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: kserve-predictor
   spec:
     action: CUSTOM
     provider:
        name: redhat-ods-applications-auth-provider 

   1

   
     rules:
        - to:
             - operation:
                  notPaths:
                     - /healthz
                     - /debug/pprof/
                     - /metrics
                     - /wait-for-drain
     selector:
        matchLabels:
           component: predictor

   Copy to Clipboard

   1

   The name that you specify must match the name of the extension provider that you added to your OpenShift Service Mesh instance.

3. Save the YAML file.

4. Create the AuthorizationPolicy resource in the namespace for your OpenShift Service Mesh instance.

   $ oc create -n <namespace_for_service_mesh_instance> -f <file_name>.yaml

   Copy to Clipboard

5. Create another new YAML file with the following contents:

   apiVersion: networking.istio.io/v1alpha3
   kind: EnvoyFilter
   metadata:
     name: activator-host-header
   spec:
     priority: 20
     workloadSelector:
       labels:
         component: predictor
     configPatches:
     - applyTo: HTTP_FILTER
       match:
         listener:
           filterChain:
             filter:
               name: envoy.filters.network.http_connection_manager
       patch:
         operation: INSERT_BEFORE
         value:
           name: envoy.filters.http.lua
           typed_config:
             '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
             inlineCode: |
              function envoy_on_request(request_handle)
                 local headers = request_handle:headers()
                 if not headers then
                   return
                 end
                 local original_host = headers:get("k-original-host")
                 if original_host then
                   port_seperator = string.find(original_host, ":", 7)
                   if port_seperator then
                     original_host = string.sub(original_host, 0, port_seperator-1)
                   end
                   headers:replace('host', original_host)
                 end
               end

   Copy to Clipboard

   The EnvoyFilter resource shown continually resets the HTTP host header to the one initially included in any inference request.

6. Create the EnvoyFilter resource in the namespace for your OpenShift Service Mesh instance.

   $ oc create -n <namespace_for_service_mesh_instance> -f <file_name>.yaml

   Copy to Clipboard