Chapter 1. Integrating Service Mesh with OpenShift Serverless

Procedure

1. Check for conflicting gateways:

   Example command

   $ oc get gateway -A -o jsonpath='{range .items[*]}{@.metadata.namespace}{"/"}{@.metadata.name}{" "}{@.spec.servers}{"\n"}{end}' | column -t

   Copy to Clipboard Toggle word wrap

   Example output

   knative-serving/knative-ingress-gateway  [{"hosts":["*"],"port":{"name":"https","number":443,"protocol":"HTTPS"},"tls":{"credentialName":"wildcard-certs","mode":"SIMPLE"}}]
   knative-serving/knative-local-gateway    [{"hosts":["*"],"port":{"name":"http","number":8081,"protocol":"HTTP"}}]

   Copy to Clipboard Toggle word wrap

   This command should not return a Gateway that binds port: 443 and hosts: ["*"], except the Gateways in knative-serving and Gateways that are part of another Service Mesh instance.

   Note

   The mesh that Serverless is part of must be distinct and preferably reserved only for Serverless workloads. That is because additional configuration, such as Gateways, might interfere with the Serverless gateways knative-local-gateway and knative-ingress-gateway. Red Hat OpenShift Service Mesh only allows one Gateway to claim a wildcard host binding (hosts: ["*"]) on the same port (port: 443). If another Gateway is already binding this configuration, a separate mesh has to be created for Serverless workloads.

2. Check whether Red Hat OpenShift Service Mesh istio-ingressgateway is exposed as type NodePort or LoadBalancer:

   Example command

   $ oc get svc -A | grep istio-ingressgateway

   Copy to Clipboard Toggle word wrap

   Example output

   istio-system   istio-ingressgateway  ClusterIP  172.30.46.146 none>   15021/TCP,80/TCP,443/TCP     9m50s

   Copy to Clipboard Toggle word wrap

   This command should not return a Service object of type NodePort or LoadBalancer.

   Note

   Cluster external Knative Services are expected to be called via OpenShift Ingress using OpenShift Routes. It is not supported to access Service Mesh directly, such as by exposing the istio-ingressgateway using a Service object with type NodePort or LoadBalancer.

Procedure

1. Create a ServiceMeshControlPlane resource in the istio-system namespace with the following configuration:

   Important

   If you have an existing ServiceMeshControlPlane object, make sure that you have the same configuration applied.

   apiVersion: maistra.io/v2
   kind: ServiceMeshControlPlane
   metadata:
     name: basic
     namespace: istio-system
   spec:
     profiles:
     - default
     security:
       dataPlane:
         mtls: true 

   1

   
     techPreview:
       meshConfig:
         defaultConfig:
           terminationDrainDuration: 35s 

   2

   
     gateways:
       ingress:
         service:
           metadata:
             labels:
               knative: ingressgateway 

   3

   
     proxy:
       networking:
         trafficControl:
           inbound:
             excludedPorts: 

   4

   
             - 8444 # metrics
             - 8022 # serving: wait-for-drain k8s pre-stop hook

   Copy to Clipboard Toggle word wrap

   1

   Enforce strict mTLS in the mesh. Only calls using a valid client certificate are allowed.

   2

   Serverless has a graceful termination for Knative Services of 30 seconds. istio-proxy needs to have a longer termination duration to make sure no requests are dropped.

   3

   Define a specific selector for the ingress gateway to target only the Knative gateway.

   4

   These ports are called by Kubernetes and cluster monitoring, which are not part of the mesh and cannot be called using mTLS. Therefore, these ports are excluded from the mesh.

2. Add the namespaces that you would like to integrate with Service Mesh to the ServiceMeshMemberRoll object as members:

   Example servicemesh-member-roll.yaml configuration file

   apiVersion: maistra.io/v1
   kind: ServiceMeshMemberRoll
   metadata:
     name: default
     namespace: istio-system
   spec:
     members: 

   1

   
       - knative-serving
       - knative-eventing
       - your-OpenShift-projects

   Copy to Clipboard Toggle word wrap

   1

   A list of namespaces to be integrated with Service Mesh.

   Important

   This list of namespaces must include the knative-serving and knative-eventing namespaces.

3. Apply the ServiceMeshMemberRoll resource:

   $ oc apply -f servicemesh-member-roll.yaml

   Copy to Clipboard Toggle word wrap

4. Create the necessary gateways so that Service Mesh can accept traffic. The following example uses the knative-local-gateway object with the ISTIO_MUTUAL mode (mTLS):

   Example istio-knative-gateways.yaml configuration file

   apiVersion: networking.istio.io/v1alpha3
   kind: Gateway
   metadata:
     name: knative-ingress-gateway
     namespace: knative-serving
   spec:
     selector:
       knative: ingressgateway
     servers:
       - port:
           number: 443
           name: https
           protocol: HTTPS
         hosts:
           - "*"
         tls:
           mode: SIMPLE
           credentialName: <wildcard_certs> 

   1

   
   ---
   apiVersion: networking.istio.io/v1alpha3
   kind: Gateway
   metadata:
    name: knative-local-gateway
    namespace: knative-serving
   spec:
    selector:
      knative: ingressgateway
    servers:
      - port:
          number: 8081
          name: https
          protocol: HTTPS 

   2

   
        tls:
          mode: ISTIO_MUTUAL 

   3

   
        hosts:
          - "*"
   ---
   apiVersion: v1
   kind: Service
   metadata:
    name: knative-local-gateway
    namespace: istio-system
    labels:
      experimental.istio.io/disable-gateway-port-translation: "true"
   spec:
    type: ClusterIP
    selector:
      istio: ingressgateway
    ports:
      - name: http2
        port: 80
        targetPort: 8081

   Copy to Clipboard Toggle word wrap

   1

   Name of the secret containing the wildcard certificate.

   2 3

   The knative-local-gateway object serves HTTPS traffic and expects all clients to send requests using mTLS. This means that only traffic coming from within Service Mesh is possible. Workloads from outside the Service Mesh must use the external domain via OpenShift Routing.

5. Apply the Gateway resources:

   $ oc apply -f istio-knative-gateways.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Install Knative Serving with the following KnativeServing custom resource, which enables the Istio integration:

   Example knative-serving-config.yaml configuration file

   apiVersion: operator.knative.dev/v1beta1
   kind: KnativeServing
   metadata:
     name: knative-serving
     namespace: knative-serving
   spec:
     ingress:
       istio:
         enabled: true 

   1

   
     deployments: 

   2

   
     - name: activator
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     - name: autoscaler
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     config:
       istio: 

   3

   
         gateway.knative-serving.knative-ingress-gateway: istio-ingressgateway.<your-istio-namespace>.svc.cluster.local
         local-gateway.knative-serving.knative-local-gateway: knative-local-gateway.<your-istio-namespace>.svc.cluster.local

   Copy to Clipboard Toggle word wrap

   1

   Enable Istio integration.

   2

   Enable sidecar injection for Knative Serving data plane pods.

   3

   If your istio is not running in the istio-system namespace, you need to set these two flags with the correct namespace.

2. Apply the KnativeServing resource:

   $ oc apply -f knative-serving-config.yaml

   Copy to Clipboard Toggle word wrap

3. Install Knative Eventing with the following KnativeEventing object, which enables the Istio integration:

   Example knative-eventing-config.yaml configuration file

   apiVersion: operator.knative.dev/v1beta1
   kind: KnativeEventing
   metadata:
     name: knative-eventing
     namespace: knative-eventing
   spec:
     config:
       features:
         istio: enabled 

   1

   
     workloads: 

   2

   
     - name: pingsource-mt-adapter
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     - name: imc-dispatcher
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     - name: mt-broker-ingress
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     - name: mt-broker-filter
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"

   Copy to Clipboard Toggle word wrap

   1

   Enable Eventing Istio controller to create a DestinationRule for each InMemoryChannel or KafkaChannel service.

   2

   Enable sidecar injection for Knative Eventing pods.

4. Apply the KnativeEventing resource:

   $ oc apply -f knative-eventing-config.yaml

   Copy to Clipboard Toggle word wrap

5. Install Knative Kafka with the following KnativeKafka custom resource, which enables the Istio integration:

   Example knative-kafka-config.yaml configuration file

   apiVersion: operator.serverless.openshift.io/v1alpha1
   kind: KnativeKafka
   metadata:
     name: knative-kafka
     namespace: knative-eventing
   spec:
     channel:
       enabled: true
       bootstrapServers: <bootstrap_servers> 

   1

   
     source:
       enabled: true
     broker:
       enabled: true
       defaultConfig:
         bootstrapServers: <bootstrap_servers> 

   2

   
         numPartitions: <num_partitions>
         replicationFactor: <replication_factor>
       sink:
         enabled: true
     workloads: 

   3

   
     - name: kafka-controller
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     - name: kafka-broker-receiver
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     - name: kafka-broker-dispatcher
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     - name: kafka-channel-receiver
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     - name: kafka-channel-dispatcher
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     - name: kafka-source-dispatcher
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"
     - name: kafka-sink-receiver
       labels:
         "sidecar.istio.io/inject": "true"
       annotations:
         "sidecar.istio.io/rewriteAppHTTPProbers": "true"

   Copy to Clipboard Toggle word wrap

   1 2

   The Apache Kafka cluster URL, for example my-cluster-kafka-bootstrap.kafka:9092.

   3

   Enable sidecar injection for Knative Kafka pods.

6. Apply the KnativeEventing object:

   $ oc apply -f knative-kafka-config.yaml

   Copy to Clipboard Toggle word wrap

7. Install ServiceEntry to inform Service Mesh of the communication between KnativeKafka components and an Apache Kafka cluster:

   Example kafka-cluster-serviceentry.yaml configuration file

   apiVersion: networking.istio.io/v1alpha3
   kind: ServiceEntry
   metadata:
     name: kafka-cluster
     namespace: knative-eventing
   spec:
     hosts: 

   1

   
       - <bootstrap_servers_without_port>
     exportTo:
       - "."
     ports: 

   2

   
       - number: 9092
         name: tcp-plain
         protocol: TCP
       - number: 9093
         name: tcp-tls
         protocol: TCP
       - number: 9094
         name: tcp-sasl-tls
         protocol: TCP
       - number: 9095
         name: tcp-sasl-tls
         protocol: TCP
       - number: 9096
         name: tcp-tls
         protocol: TCP
     location: MESH_EXTERNAL
     resolution: NONE

   Copy to Clipboard Toggle word wrap

   1

   The list of Apache Kafka cluster hosts, for example my-cluster-kafka-bootstrap.kafka.

   2

   Apache Kafka cluster listeners ports.

   Note

   The listed ports in spec.ports are example TPC ports. The actual values depend on how the Apache Kafka cluster is configured.

8. Apply the ServiceEntry resource:

   $ oc apply -f kafka-cluster-serviceentry.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a Knative Service that has sidecar injection enabled and uses a pass-through route:

   Example knative-service.yaml configuration file

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: <service_name>
     namespace: <namespace> 

   1

   
     annotations:
       serving.knative.openshift.io/enablePassthrough: "true" 

   2

   
   spec:
     template:
       metadata:
         annotations:
           sidecar.istio.io/inject: "true" 

   3

   
           sidecar.istio.io/rewriteAppHTTPProbers: "true"
       spec:
         containers:
         - image: <image_url>

   Copy to Clipboard Toggle word wrap

   1

   A namespace that is part of the service mesh member roll.

   2

   Instruct Knative Serving to generate a pass-through enabled route, so that the certificates you have generated are served through the ingress gateway directly.

   3

   Inject Service Mesh sidecars into the Knative service pods.

   Important

   Always add the annotation from this example to all of your Knative Service to make them work with Service Mesh.

2. Apply the Service resource:

   $ oc apply -f knative-service.yaml

   Copy to Clipboard Toggle word wrap

3. Access your serverless application by using a secure connection that is now trusted by the CA:

   $ curl --cacert root.crt <service_url>

   Copy to Clipboard Toggle word wrap

   For example, run:

   Example command

   $ curl --cacert root.crt https://hello-default.apps.openshift.example.com

   Copy to Clipboard Toggle word wrap

   Example output

   Hello Openshift!

   Copy to Clipboard Toggle word wrap