Chapter 3. Serving large models

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

2. Create the required namespace for Red Hat OpenShift Service Mesh.

   $ oc create ns istio-system

   You see the following output:

   namespace/istio-system created

3. Define a ServiceMeshControlPlane object in a YAML file named smcp.yaml with the following contents:

   apiVersion: maistra.io/v2
   kind: ServiceMeshControlPlane
   metadata:
     name: minimal
     namespace: istio-system
   spec:
     tracing:
       type: None
     addons:
       grafana:
         enabled: false
       kiali:
         name: kiali
         enabled: false
       prometheus:
         enabled: false
       jaeger:
         name: jaeger
     security:
       dataPlane:
         mtls: true
       identity:
         type: ThirdParty
     techPreview:
       meshConfig:
         defaultConfig:
           terminationDrainDuration: 35s
     gateways:
       ingress:
         service:
           metadata:
             labels:
               knative: ingressgateway
     proxy:
       networking:
         trafficControl:
           inbound:
             excludedPorts:
               - 8444
               - 8022

   For more information about the values in the YAML file, see the Service Mesh control plane configuration reference.

4. Create the service mesh control plane.

   $ oc apply -f smcp.yaml

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

2. Check whether the required project (that is, namespace) for Knative Serving already exists.

   $ oc get ns knative-serving

   If the project exists, you see output similar to the following example:

   NAME              STATUS   AGE
   knative-serving   Active   4d20h

3. If the knative-serving project doesn’t already exist, create it.

   $ oc create ns knative-serving

   You see the following output:

   namespace/knative-serving created

4. Define a ServiceMeshMember object in a YAML file called default-smm.yaml with the following contents:

   apiVersion: maistra.io/v1
   kind: ServiceMeshMember
   metadata:
     name: default
     namespace: knative-serving
   spec:
     controlPlaneRef:
       namespace: istio-system
       name: minimal

5. Create the ServiceMeshMember object in the istio-system namespace.

   $ oc apply -f default-smm.yaml

   You see the following output:

   servicemeshmember.maistra.io/default created

6. Define a KnativeServing object in a YAML file called knativeserving-istio.yaml with the following contents:

   apiVersion: operator.knative.dev/v1beta1
   kind: KnativeServing
   metadata:
     name: knative-serving
     namespace: knative-serving
     annotations:
       serverless.openshift.io/default-enable-http2: "true"
   spec:
     workloads:
       - name: net-istio-controller
         env:
           - container: controller
             envVars:
               - name: ENABLE_SECRET_INFORMER_FILTERING_BY_CERT_UID
                 value: 'true'
       - annotations:
           sidecar.istio.io/inject: "true" 1
           sidecar.istio.io/rewriteAppHTTPProbers: "true" 2
         name: activator
       - annotations:
           sidecar.istio.io/inject: "true"
           sidecar.istio.io/rewriteAppHTTPProbers: "true"
         name: autoscaler
     ingress:
       istio:
         enabled: true
     config:
       features:
         kubernetes.podspec-affinity: enabled
         kubernetes.podspec-nodeselector: enabled
         kubernetes.podspec-tolerations: enabled

   The preceding file defines a custom resource (CR) for a KnativeServing object. The CR also adds the following actions to each of the activator and autoscaler pods:

   1

   Injects an Istio sidecar to the pod. This makes the pod part of the service mesh.

   2

   Enables the Istio sidecar to rewrite the HTTP liveness and readiness probes for the pod.

   Note

   If you configure a custom domain for a Knative service, you can use a TLS certificate to secure the mapped service. To do this, you must create a TLS secret, and then update the DomainMapping CR to use the TLS secret that you have created. For more information, see Securing a mapped service using a TLS certificate in the Red Hat OpenShift Serverless documentation.

7. Create the KnativeServing object in the specified knative-serving namespace.

   $ oc apply -f knativeserving-istio.yaml

   You see the following output:

   knativeserving.operator.knative.dev/knative-serving created

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Important

   If you have your own wildcard certificate and key to specify when configuring the gateways, skip to step 11 of this procedure.

2. Set environment variables to define base directories for generation of a wildcard certificate and key for the gateways.

   $ export BASE_DIR=/tmp/kserve
   $ export BASE_CERT_DIR=${BASE_DIR}/certs

3. Set an environment variable to define the common name used by the ingress controller of your OpenShift cluster.

   $ export COMMON_NAME=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}' | awk -F'.' '{print $(NF-1)"."$NF}')

4. Set an environment variable to define the domain name used by the ingress controller of your OpenShift cluster.

   $ export DOMAIN_NAME=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}')

5. Create the required base directories for the certificate generation, based on the environment variables that you previously set.

   $ mkdir ${BASE_DIR}
   $ mkdir ${BASE_CERT_DIR}

6. Create the OpenSSL configuration for generation of a wildcard certificate.

   $ cat <<EOF> ${BASE_DIR}/openssl-san.config
   [ req ]
   distinguished_name = req
   [ san ]
   subjectAltName = DNS:*.${DOMAIN_NAME}
   EOF

7. Generate a root certificate.

   $ openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 \
   -subj "/O=Example Inc./CN=${COMMON_NAME}" \
   -keyout $BASE_DIR/root.key \
   -out $BASE_DIR/root.crt

8. Generate a wildcard certificate signed by the root certificate.

   $ openssl req -x509 -newkey rsa:2048 \
   -sha256 -days 3560 -nodes \
   -subj "/CN=${COMMON_NAME}/O=Example Inc." \
   -extensions san -config ${BASE_DIR}/openssl-san.config \
   -CA $BASE_DIR/root.crt \
   -CAkey $BASE_DIR/root.key \
   -keyout $BASE_DIR/wildcard.key  \
   -out $BASE_DIR/wildcard.crt
   
   $ openssl x509 -in ${BASE_DIR}/wildcard.crt -text

9. Verify the wildcard certificate.

   $ openssl verify -CAfile ${BASE_DIR}/root.crt ${BASE_DIR}/wildcard.crt

10. Export the wildcard key and certificate that were created by the script to new environment variables.

    $ export TARGET_CUSTOM_CERT=${BASE_CERT_DIR}/wildcard.crt
    $ export TARGET_CUSTOM_KEY=${BASE_CERT_DIR}/wildcard.key

11. Optional: To export your own wildcard key and certificate to new environment variables, enter the following commands:

    $ export TARGET_CUSTOM_CERT=<path_to_certificate>
    $ export TARGET_CUSTOM_KEY=<path_to_key>

    Note

    In the certificate that you provide, you must specify the domain name used by the ingress controller of your OpenShift cluster. You can check this value by running the following command:

    $ oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}'

12. Create a TLS secret in the istio-system namespace using the environment variables that you set for the wildcard certificate and key.

    $ oc create secret tls wildcard-certs --cert=${TARGET_CUSTOM_CERT} --key=${TARGET_CUSTOM_KEY} -n istio-system

13. Create a gateways.yaml YAML file with the following contents:

    apiVersion: v1
    kind: Service 1
    metadata:
      labels:
        experimental.istio.io/disable-gateway-port-translation: "true"
      name: knative-local-gateway
      namespace: istio-system
    spec:
      ports:
        - name: http2
          port: 80
          protocol: TCP
          targetPort: 8081
      selector:
        knative: ingressgateway
      type: ClusterIP
    ---
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
      name: knative-ingress-gateway 2
      namespace: knative-serving
    spec:
      selector:
        knative: ingressgateway
      servers:
        - hosts:
            - '*'
          port:
            name: https
            number: 443
            protocol: HTTPS
          tls:
            credentialName: wildcard-certs
            mode: SIMPLE
    ---
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
     name: knative-local-gateway 3
     namespace: knative-serving
    spec:
     selector:
       knative: ingressgateway
     servers:
       - port:
           number: 8081
           name: https
           protocol: HTTPS
         tls:
           mode: ISTIO_MUTUAL
         hosts:
           - "*"

    1

    Defines a service in the istio-system namespace for the Knative local gateway.

    2

    Defines an ingress gateway in the knative-serving namespace. The gateway uses the TLS secret you created earlier in this procedure. The ingress gateway handles external traffic to Knative.

    3

    Defines a local gateway for Knative in the knative-serving namespace.

14. Apply the gateways.yaml file to create the defined resources.

    $ oc apply -f gateways.yaml

    You see the following output:

    service/knative-local-gateway created
    gateway.networking.istio.io/knative-ingress-gateway created
    gateway.networking.istio.io/knative-local-gateway created

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators OperatorHub.
3. On the OperatorHub page, in the Filter by keyword field, type Red Hat - Authorino.
4. Click the Red Hat - Authorino Operator.
5. On the Red Hat - Authorino Operator page, review the Operator information and then click Install.
6. On the Install Operator page, keep the default values for Update channel, Version, Installation mode, Installed Namespace and Update Approval.
7. Click Install.

Procedure

1. Open a new terminal window.

2. Log in to the OpenShift command-line interface (CLI) as follows:

   $ oc login <openshift_cluster_url> -u <username> -p <password>

3. Create a namespace to install the Authorino instance.

   $ oc new-project <namespace_for_authorino_instance>

   Note

   The automated installation process creates a namespace called redhat-ods-applications-auth-provider for the Authorino instance. Consider using the same namespace name for the manual installation.

4. To enroll the new namespace for the Authorino instance in your existing OpenShift Service Mesh instance, create a new YAML file with the following contents:

     apiVersion: maistra.io/v1
     kind: ServiceMeshMember
     metadata:
       name: default
       namespace: <namespace_for_authorino_instance>
     spec:
       controlPlaneRef:
         namespace: <namespace_for_service_mesh_instance>
         name: <name_of_service_mesh_instance>

5. Save the YAML file.

6. Create the ServiceMeshMember resource on your cluster.

   $ oc create -f <file_name>.yaml

7. To configure an Authorino instance, create a new YAML file as shown in the following example:

     apiVersion: operator.authorino.kuadrant.io/v1beta1
     kind: Authorino
     metadata:
       name: authorino
       namespace: <namespace_for_authorino_instance>
     spec:
       authConfigLabelSelectors: security.opendatahub.io/authorization-group=default
       clusterWide: true
       listener:
         tls:
           enabled: false
       oidcServer:
         tls:
           enabled: false

8. Save the YAML file.

9. Create the Authorino resource on your cluster.

   $ oc create -f <file_name>.yaml

10. Patch the Authorino deployment to inject an Istio sidecar, which makes the Authorino instance part of your OpenShift Service Mesh instance.

    $ oc patch deployment <name_of_authorino_instance> -n <namespace_for_authorino_instance> -p '{"spec": {"template":{"metadata":{"labels":{"sidecar.istio.io/inject":"true"}}}} }'

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a user that has privileges to update the OpenShift Service Mesh instance, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <username> -p <password>

2. Create a new YAML file with the following contents:

   spec:
    techPreview:
      meshConfig:
        extensionProviders:
        - name: redhat-ods-applications-auth-provider
          envoyExtAuthzGrpc:
            service: <name_of_authorino_instance>-authorino-authorization.<namespace_for_authorino_instance>.svc.cluster.local
            port: 50051

3. Save the YAML file.

4. Use the oc patch command to apply the YAML file to your OpenShift Service Mesh instance.

   $ oc patch smcp <name_of_service_mesh_instance> --type merge -n <namespace_for_service_mesh_instance> --patch-file <file_name>.yaml

   Important

   You can apply the configuration shown as a patch only if you have not already specified other extension providers in your OpenShift Service Mesh instance. If you have already specified other extension providers, you must manually edit your ServiceMeshControlPlane resource to add the configuration.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a user that has privileges to update the KServe deployment, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <username> -p <password>

2. Create a new YAML file with the following contents:

   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: kserve-predictor
   spec:
     action: CUSTOM
     provider:
        name: redhat-ods-applications-auth-provider 1
     rules:
        - to:
             - operation:
                  notPaths:
                     - /healthz
                     - /debug/pprof/
                     - /metrics
                     - /wait-for-drain
     selector:
        matchLabels:
           component: predictor

   1

   The name that you specify must match the name of the extension provider that you added to your OpenShift Service Mesh instance.

3. Save the YAML file.

4. Create the AuthorizationPolicy resource in the namespace for your OpenShift Service Mesh instance.

   $ oc create -n <namespace_for_service_mesh_instance> -f <file_name>.yaml

5. Create another new YAML file with the following contents:

   apiVersion: networking.istio.io/v1alpha3
   kind: EnvoyFilter
   metadata:
     name: activator-host-header
   spec:
     priority: 20
     workloadSelector:
       labels:
         component: predictor
     configPatches:
     - applyTo: HTTP_FILTER
       match:
         listener:
           filterChain:
             filter:
               name: envoy.filters.network.http_connection_manager
       patch:
         operation: INSERT_BEFORE
         value:
           name: envoy.filters.http.lua
           typed_config:
             '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
             inlineCode: |
              function envoy_on_request(request_handle)
                 local headers = request_handle:headers()
                 if not headers then
                   return
                 end
                 local original_host = headers:get("k-original-host")
                 if original_host then
                   port_seperator = string.find(original_host, ":", 7)
                   if port_seperator then
                     original_host = string.sub(original_host, 0, port_seperator-1)
                   end
                   headers:replace('host', original_host)
                 end
               end

   The EnvoyFilter resource shown continually resets the HTTP host header to the one initially included in any inference request.

6. Create the EnvoyFilter resource in the namespace for your OpenShift Service Mesh instance.

   $ oc create -n <namespace_for_service_mesh_instance> -f <file_name>.yaml