Chapter 2. Using Service Mesh to isolate network traffic with OpenShift Serverless

Procedure

1. Make sure that all Red Hat OpenShift Serverless projects of your tenant are part of the same ServiceMeshMemberRoll object as members:

   apiVersion: maistra.io/v1
   kind: ServiceMeshMemberRoll
   metadata:
    name: default
    namespace: istio-system
   spec:
    members:
      - knative-serving    # static value, needs to be here, see setup page
      - knative-eventing   # static value, needs to be here, see setup page
      - team-alpha-1       # example OpenShift project that belongs to the team-alpha tenant
      - team-alpha-2       # example OpenShift project that belongs th the team-alpha tenant
      - team-bravo-1       # example OpenShift project that belongs to the team-bravo tenant
      - team-bravo-2       # example OpenShift project that belongs th the team-bravo tenant

   All projects that are part of the mesh must enforce mTLS in strict mode. This forces Istio to only accept connections with a client-certificate present and allows the Service Mesh sidecar to validate the origin using an AuthorizationPolicy object.

2. Create the configuration with AuthorizationPolicy objects in the knative-serving and knative-eventing namespaces:

   Example knative-default-authz-policies.yaml configuration file

   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: deny-all-by-default
     namespace: knative-eventing
   spec: { }
   ---
   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: deny-all-by-default
     namespace: knative-serving
   spec: { }
   ---
   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: allow-mt-channel-based-broker-ingress-to-imc-dispatcher
     namespace: knative-eventing
   spec:
     action: ALLOW
     selector:
       matchLabels:
         app.kubernetes.io/component: "imc-dispatcher"
     rules:
       - from:
           - source:
               namespaces: [ "knative-eventing" ]
               principals: [ "cluster.local/ns/knative-eventing/sa/mt-broker-ingress" ]
         to:
           - operation:
               methods: [ "POST" ]
   ---
   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: allow-mt-channel-based-broker-ingress-to-kafka-channel
     namespace: knative-eventing
   spec:
     action: ALLOW
     selector:
       matchLabels:
         app.kubernetes.io/component: "kafka-channel-receiver"
     rules:
       - from:
           - source:
               namespaces: [ "knative-eventing" ]
               principals: [ "cluster.local/ns/knative-eventing/sa/mt-broker-ingress" ]
         to:
           - operation:
               methods: [ "POST" ]
   ---
   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: allow-kafka-channel-to-mt-channel-based-broker-filter
     namespace: knative-eventing
   spec:
     action: ALLOW
     selector:
       matchLabels:
         app.kubernetes.io/component: "broker-filter"
     rules:
       - from:
           - source:
               namespaces: [ "knative-eventing" ]
               principals: [ "cluster.local/ns/knative-eventing/sa/knative-kafka-channel-data-plane" ]
         to:
           - operation:
               methods: [ "POST" ]
   ---
   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: allow-imc-to-mt-channel-based-broker-filter
     namespace: knative-eventing
   spec:
     action: ALLOW
     selector:
       matchLabels:
         app.kubernetes.io/component: "broker-filter"
     rules:
       - from:
           - source:
               namespaces: [ "knative-eventing" ]
               principals: [ "cluster.local/ns/knative-eventing/sa/imc-dispatcher" ]
         to:
           - operation:
               methods: [ "POST" ]
   ---
   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: allow-probe-kafka-broker-receiver
     namespace: knative-eventing
   spec:
     action: ALLOW
     selector:
       matchLabels:
         app.kubernetes.io/component: "kafka-broker-receiver"
     rules:
       - from:
           - source:
               namespaces: [ "knative-eventing" ]
               principals: [ "cluster.local/ns/knative-eventing/sa/kafka-controller" ]
         to:
           - operation:
               methods: [ "GET" ]
   ---
   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: allow-probe-kafka-sink-receiver
     namespace: knative-eventing
   spec:
     action: ALLOW
     selector:
       matchLabels:
         app.kubernetes.io/component: "kafka-sink-receiver"
     rules:
       - from:
           - source:
               namespaces: [ "knative-eventing" ]
               principals: [ "cluster.local/ns/knative-eventing/sa/kafka-controller" ]
         to:
           - operation:
               methods: [ "GET" ]
   ---
   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: allow-probe-kafka-channel-receiver
     namespace: knative-eventing
   spec:
     action: ALLOW
     selector:
       matchLabels:
         app.kubernetes.io/component: "kafka-channel-receiver"
     rules:
       - from:
           - source:
               namespaces: [ "knative-eventing" ]
               principals: [ "cluster.local/ns/knative-eventing/sa/kafka-controller" ]
         to:
           - operation:
               methods: [ "GET" ]
   ---
   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: allow-traffic-to-activator
     namespace: knative-serving
   spec:
     selector:
       matchLabels:
         app: activator
     action: ALLOW
     rules:
       - from:
           - source:
               namespaces: [ "knative-serving", "istio-system" ]
   ---
   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: allow-traffic-to-autoscaler
     namespace: knative-serving
   spec:
     selector:
       matchLabels:
         app: autoscaler
     action: ALLOW
     rules:
       - from:
           - source:
               namespaces: [ "knative-serving" ]

   These policies restrict the access rules for the network communication between Serverless system components. Specifically, they enforce the following rules:

   • Deny all traffic that is not explicitly allowed in the knative-serving and knative-eventing namespaces
   • Allow traffic from the istio-system and knative-serving namespaces to activator
   • Allow traffic from the knative-serving namespace to autoscaler
   • Allow health probes for Apache Kafka components in the knative-eventing namespace
   • Allow internal traffic for channel-based brokers in the knative-eventing namespace

3. Apply the authorization policy configuration:

   $ oc apply -f knative-default-authz-policies.yaml

4. Define which OpenShift projects can communicate with each other. For this communication, every OpenShift project of a tenant requires the following:

   • One AuthorizationPolicy object limiting directly incoming traffic to the tenant’s project
   • One AuthorizationPolicy object limiting incoming traffic using the activator component of Serverless that runs in the knative-serving project
   • One AuthorizationPolicy object allowing Kubernetes to call PreStopHooks on Knative Services

   Instead of creating these policies manually, install the helm utility and create the necessary resources for each tenant:

   Installing the helm utility

   $ helm repo add openshift-helm-charts https://charts.openshift.io/

   Creating example configuration for team alpha

   $ helm template openshift-helm-charts/redhat-knative-istio-authz --version 1.31.0 --set "name=team-alpha" --set "namespaces={team-alpha-1,team-alpha-2}" > team-alpha.yaml

   Creating example configuration for team bravo

   $ helm template openshift-helm-charts/redhat-knative-istio-authz --version 1.31.0 --set "name=team-bravo" --set "namespaces={team-bravo-1,team-bravo-2}" > team-bravo.yaml

5. Apply the authorization policy configuration:

   $ oc apply -f team-alpha.yaml team-bravo.yaml

Procedure

1. Deploy Knative Services in the namespaces of both of the tenants:

   Example command for team-alpha

   $ kn service create test-webapp -n team-alpha-1 \
       --annotation-service serving.knative.openshift.io/enablePassthrough=true \
       --annotation-revision sidecar.istio.io/inject=true \
       --env RESPONSE="Hello Serverless" \
       --image docker.io/openshift/hello-openshift

   Example command for team-bravo

   $ kn service create test-webapp -n team-bravo-1 \
       --annotation-service serving.knative.openshift.io/enablePassthrough=true \
       --annotation-revision sidecar.istio.io/inject=true \
       --env RESPONSE="Hello Serverless" \
       --image docker.io/openshift/hello-openshift

   Alternatively, use the following YAML configuration:

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: test-webapp
     namespace: team-alpha-1
     annotations:
       serving.knative.openshift.io/enablePassthrough: "true"
   spec:
     template:
       metadata:
         annotations:
           sidecar.istio.io/inject: 'true'
       spec:
         containers:
           - image: docker.io/openshift/hello-openshift
             env:
               - name: RESPONSE
                 value: "Hello Serverless!"
   ---
   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: test-webapp
     namespace: team-bravo-1
     annotations:
       serving.knative.openshift.io/enablePassthrough: "true"
   spec:
     template:
       metadata:
         annotations:
           sidecar.istio.io/inject: 'true'
       spec:
         containers:
           - image: docker.io/openshift/hello-openshift
             env:
               - name: RESPONSE
                 value: "Hello Serverless!"

2. Deploy a curl pod for testing the connections:

   $ cat <<EOF | oc apply -f -
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: curl
     namespace: team-alpha-1
     labels:
       app: curl
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: curl
     template:
       metadata:
         labels:
           app: curl
         annotations:
           sidecar.istio.io/inject: 'true'
       spec:
         containers:
         - name: curl
           image: curlimages/curl
           command:
           - sleep
           - "3600"
   EOF

3. Verify the configuration by using the curl command.

   Test team-alpha-1 team-alpha-1 through cluster local domain, which is allowed:

   Example command

   $ oc exec deployment/curl -n team-alpha-1 -it -- curl -v http://test-webapp.team-alpha-1:80

   Example output

   HTTP/1.1 200 OK
   content-length: 18
   content-type: text/plain; charset=utf-8
   date: Wed, 26 Jul 2023 12:49:59 GMT
   server: envoy
   x-envoy-upstream-service-time: 9
   
   Hello Serverless!

   Test the team-alpha-1 to team-alpha-1 connection through an external domain, which is allowed:

   Example command

   $ EXTERNAL_URL=$(oc get ksvc -n team-alpha-1 test-webapp -o custom-columns=:.status.url --no-headers) && \
   oc exec deployment/curl -n team-alpha-1 -it -- curl -ik $EXTERNAL_URL

   Example output

   HTTP/2 200
   content-length: 18
   content-type: text/plain; charset=utf-8
   date: Wed, 26 Jul 2023 12:55:30 GMT
   server: istio-envoy
   x-envoy-upstream-service-time: 3629
   
   Hello Serverless!

   Test the team-alpha-1 to team-bravo-1 connection through the cluster’s local domain, which is not allowed:

   Example command

   $ oc exec deployment/curl -n team-alpha-1 -it -- curl -v http://test-webapp.team-bravo-1:80

   Example output

   * processing: http://test-webapp.team-bravo-1:80
   *   Trying 172.30.73.216:80...
   * Connected to test-webapp.team-bravo-1 (172.30.73.216) port 80
   > GET / HTTP/1.1
   > Host: test-webapp.team-bravo-1
   > User-Agent: curl/8.2.0
   > Accept: */*
   >
   < HTTP/1.1 403 Forbidden
   < content-length: 19
   < content-type: text/plain
   < date: Wed, 26 Jul 2023 12:55:49 GMT
   < server: envoy
   < x-envoy-upstream-service-time: 6
   <
   * Connection #0 to host test-webapp.team-bravo-1 left intact
   RBAC: access denied

   Test the team-alpha-1 to team-bravo-1 connection through an external domain, which is allowed:

   Example command

   $ EXTERNAL_URL=$(oc get ksvc -n team-bravo-1 test-webapp -o custom-columns=:.status.url --no-headers) && \
   oc exec deployment/curl -n team-alpha-1 -it -- curl -ik $EXTERNAL_URL

   Example output

   HTTP/2 200
   content-length: 18
   content-type: text/plain; charset=utf-8
   date: Wed, 26 Jul 2023 12:56:22 GMT
   server: istio-envoy
   x-envoy-upstream-service-time: 2856
   
   Hello Serverless!

4. Delete the resources that were created for verification:

   $ oc delete deployment/curl -n team-alpha-1 && \
   oc delete ksvc/test-webapp -n team-alpha-1 && \
   oc delete ksvc/test-webapp -n team-bravo-1