Chapter 6. Serving large language models

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

2. Create the required namespace for Red Hat OpenShift Service Mesh.

   $ oc create ns istio-system

   You see the following output:

   namespace/istio-system created

3. Define a ServiceMeshControlPlane object in a YAML file named smcp.yaml with the following contents:

   apiVersion: maistra.io/v2
   kind: ServiceMeshControlPlane
   metadata:
     name: minimal
     namespace: istio-system
   spec:
     tracing:
       type: None
     addons:
       grafana:
         enabled: false
       kiali:
         name: kiali
         enabled: false
       prometheus:
         enabled: false
       jaeger:
         name: jaeger
     security:
       dataPlane:
         mtls: true
       identity:
         type: ThirdParty
     techPreview:
       meshConfig:
         defaultConfig:
           terminationDrainDuration: 35s
     gateways:
       ingress:
         service:
           metadata:
             labels:
               knative: ingressgateway
     proxy:
       networking:
         trafficControl:
           inbound:
             excludedPorts:
               - 8444
               - 8022

   For more information about the values in the YAML file, see the Service Mesh control plane configuration reference.

4. Create the service mesh control plane.

   $ oc apply -f smcp.yaml

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

2. Check whether the required project (that is, namespace) for Knative Serving already exists.

   $ oc get ns knative-serving

   If the project exists, you see output similar to the following example:

   NAME              STATUS   AGE
   knative-serving   Active   4d20h

3. If the knative-serving project doesn’t already exist, create it.

   $ oc create ns knative-serving

   You see the following output:

   namespace/knative-serving created

4. Define a ServiceMeshMember object in a YAML file called default-smm.yaml with the following contents:

   apiVersion: maistra.io/v1
   kind: ServiceMeshMember
   metadata:
     name: default
     namespace: knative-serving
   spec:
     controlPlaneRef:
       namespace: istio-system
       name: minimal

5. Create the ServiceMeshMember object in the istio-system namespace.

   $ oc apply -f default-smm.yaml

   You see the following output:

   servicemeshmember.maistra.io/default created

6. Define a KnativeServing object in a YAML file called knativeserving-istio.yaml with the following contents:

   apiVersion: operator.knative.dev/v1beta1
   kind: KnativeServing
   metadata:
     name: knative-serving
     namespace: knative-serving
     annotations:
       serverless.openshift.io/default-enable-http2: "true"
   spec:
     deployments:
       - annotations:
           sidecar.istio.io/inject: "true" 1
           sidecar.istio.io/rewriteAppHTTPProbers: "true 2
         name: activator
       - annotations:
           sidecar.istio.io/inject: "true"
           sidecar.istio.io/rewriteAppHTTPProbers: "true"
         name: autoscaler
     ingress:
       istio:
         enabled: true
     config:
       features:
         kubernetes.podspec-affinity: enabled
         kubernetes.podspec-nodeselector: enabled
         kubernetes.podspec-tolerations: enabled

   The preceding file defines a custom resource (CR) for a KnativeServing object. The CR also adds the following actions to each of the activator and autoscaler pods:

   1

   Injects an Isitio sidecar to the pod. This makes the pod part of the service mesh.

   2

   Enables the Istio sidecar to rewrite the HTTP liveness and readiness probes for the pod.

7. Create the KnativeServing object in the specified knative-serving namespace.

   $ oc apply -f knativeserving-istio.yaml

   You see the following output:

   knativeserving.operator.knative.dev/knative-serving created

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Important

   If you have your own wildcard certificate and key to specify when configuring the gateways, skip to step 11 of this procedure.

2. Set environment variables to define base directories for generation of a wildcard certificate and key for the gateways.

   $ export BASE_DIR=/tmp/kserve
   $ export BASE_CERT_DIR=${BASE_DIR}/certs

3. Set an environment variable to define the common name used by the ingress controller of your OpenShift cluster.

   $ export COMMON_NAME=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}' | awk -F'.' '{print $(NF-1)"."$NF}')

4. Set an environment variable to define the domain name used by the ingress controller of your OpenShift cluster.

   $ export DOMAIN_NAME=$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}')

5. Create the required base directories for the certificate generation, based on the environment variables that you previously set.

   $ mkdir ${BASE_DIR}
   $ mkdir ${BASE_CERT_DIR}

6. Create the OpenSSL configuration for generation of a wildcard certificate.

   $ cat <<EOF> ${BASE_DIR}/openssl-san.config
   [ req ]
   distinguished_name = req
   [ san ]
   subjectAltName = DNS:*.${DOMAIN_NAME}
   EOF

7. Generate a root certificate.

   $ openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 \
   -subj "/O=Example Inc./CN=${COMMON_NAME}" \
   -keyout $BASE_DIR/root.key \
   -out $BASE_DIR/root.crt

8. Generate a wildcard certificate signed by the root certificate.

   $ openssl req -x509 -newkey rsa:2048 \
   -sha256 -days 3560 -nodes \
   -subj "/CN=${COMMON_NAME}/O=Example Inc." \
   -extensions san -config ${BASE_DIR}/openssl-san.config \
   -CA $BASE_DIR/root.crt \
   -CAkey $BASE_DIR/root.key \
   -keyout $BASE_DIR/wildcard.key  \
   -out $BASE_DIR/wildcard.crt
   
   $ openssl x509 -in ${BASE_DIR}/wildcard.crt -text

9. Verify the wildcard certificate.

   $ openssl verify -CAfile ${BASE_DIR}/root.crt ${BASE_DIR}/wildcard.crt

10. Export the wildcard key and certificate that were created by the script to new environment variables.

    $ export TARGET_CUSTOM_CERT=${BASE_CERT_DIR}/wildcard.crt
    $ export TARGET_CUSTOM_KEY=${BASE_CERT_DIR}/wildcard.key

11. Optional: To export your own wildcard key and certificate to new environment variables, enter the following commands:

    $ export TARGET_CUSTOM_CERT=<path_to_certificate>
    $ export TARGET_CUSTOM_KEY=<path_to_key>

    Note

    In the certificate that you provide, you must specify the domain name used by the ingress controller of your OpenShift cluster. You can check this value by running the following command:

    $ oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}'

12. Create a TLS secret in the istio-system namespace using the environment variables that you set for the wildcard certificate and key.

    $ oc create secret tls wildcard-certs --cert=${TARGET_CUSTOM_CERT} --key=${TARGET_CUSTOM_KEY} -n istio-system

13. Create a gateways.yaml YAML file with the following contents:

    apiVersion: v1
    kind: Service 1
    metadata:
      labels:
        experimental.istio.io/disable-gateway-port-translation: "true"
      name: knative-local-gateway
      namespace: istio-system
    spec:
      ports:
        - name: http2
          port: 80
          protocol: TCP
          targetPort: 8081
      selector:
        knative: ingressgateway
      type: ClusterIP
    ---
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
      name: knative-ingress-gateway 2
      namespace: knative-serving
    spec:
      selector:
        knative: ingressgateway
      servers:
        - hosts:
            - '*'
          port:
            name: https
            number: 443
            protocol: HTTPS
          tls:
            credentialName: wildcard-certs
            mode: SIMPLE
    ---
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
     name: knative-local-gateway 3
     namespace: knative-serving
    spec:
     selector:
       knative: ingressgateway
     servers:
       - port:
           number: 8081
           name: https
           protocol: HTTPS
         tls:
           mode: ISTIO_MUTUAL
         hosts:
           - "*"

    1

    Defines a service in the istio-system namespace for the Knative local gateway.

    2

    Defines an ingress gateway in the knative-serving namespace. The gateway uses the TLS secret you created earlier in this procedure. The ingress gateway handles external traffic to Knative.

    3

    Defines a local gateway for Knative in the knative-serving namespace.

14. Apply the gateways.yaml file to create the defined resources.

    $ oc apply -f gateways.yaml

    You see the following output:

    service/knative-local-gateway created
    gateway.networking.istio.io/knative-ingress-gateway created
    gateway.networking.istio.io/knative-local-gateway created