Chapter 5. OpenShift Service Mesh and cert-manager

Procedure

1. Create the istio-system namespace by running the following command:

   $ oc create namespace istio-system

2. Create the root certificate authority (CA) issuer by creating an Issuer object for the istio-csr agent:

   1. Create a new project to install the istio-csr agent by running the following command:

      $ oc new-project istio-csr

   2. Define an Issuer object similar to the following example:

      Note

      The selfSigned issuer serves demonstration purposes, testing, or proof-of-concept environments. For production deployments, use a secure and trusted CA.

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: selfsigned
        namespace: istio-system
      spec:
        selfSigned: {}
      ---
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: istio-ca
        namespace: istio-system
      spec:
        isCA: true
        duration: 87600h
        secretName: istio-ca
        commonName: istio-ca
        privateKey:
          algorithm: ECDSA
          size: 256
        subject:
          organizations:
            - cluster.local
            - cert-manager
        issuerRef:
          name: selfsigned
          kind: Issuer
          group: cert-manager.io
      ---
      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: istio-ca
        namespace: istio-system
      spec:
        ca:
          secretName: istio-ca

   3. Create the objects by running the following command:

      $ oc apply -f issuer.yaml

   4. Wait for the istio-ca certificate to contain the "Ready" status condition by running the following command:

      $ oc wait --for=condition=Ready certificates/istio-ca -n istio-system

3. Deploy the istio-csr agent:

   1. Define the IstioCSR custom resource as shown in the following example:

      apiVersion: operator.openshift.io/v1alpha1
      kind: IstioCSR
      metadata:
        name: default
        namespace: istio-csr
      spec:
        istioCSRConfig:
          certManager:
            issuerRef:
              name: istio-ca
              kind: Issuer
              group: cert-manager.io
          istiodTLSConfig:
            trustDomain: cluster.local
          istio:
            namespace: istio-system

   2. Run the following command to apply the custom resource configuration and deploy the istio-csr agent:

      $ oc create -f istioCSR.yaml

   3. Verify that the istio-csr deployment is ready by running the following command:

      $ oc get deployment -n istio-csr

4. Install the Istio resource:

   Note

   The configuration disables the built-in CA server for Istio and forwards certificate signing requests from istiod to the istio-csr agent. The istio-csr agent obtains certificates for both istiod and mesh workloads from the cert-manager Operator. The istio-csr agent generates the istiod TLS certificate, and the system mounts it into the pod at a known location.

   1. Define the istio object to delegate certificate signing to the istio-csr agent. Refer to the following example:

      apiVersion: sailoperator.io/v1
      kind: Istio
      metadata:
        name: default
      spec:
        namespace: istio-system
        values:
          global:
            caAddress: cert-manager-istio-csr.istio-csr.svc:443

      Note

      If you use Istio ambient mode, remember to add the previous caAddress configuration to your istiod configuration.

   2. Create the Istio resource by running the following command:

      $ oc apply -f istio.yaml

   3. Verify that the Istio resource displays the "Ready" status condition by running the following command:

      $ oc wait --for=condition=Ready istios/default -n istio-system

Procedure

1. Create the followingnamespaces:

   1. Create the apps-1 namespace by running the following command:

      $ oc new-project apps-1

   2. Create the apps-2 namespace by running the following command:

      $ oc new-project apps-2

2. Add the istio-injection=enabled label on the namespaces:

   1. Add the istio-injection=enabled label on the apps-1 namespace by running the following command:

      $ oc label namespaces apps-1 istio-injection=enabled

   2. Add the istio-injection=enabled label on the apps-2 namespace by running the following command:

      $ oc label namespaces apps-2 istio-injection=enabled

3. Deploy the httpbin app in the namespaces:

   1. Deploy the httpbin app in the apps-1 namespace by running the following command:

      $ oc apply -n apps-1 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/httpbin/httpbin.yaml

   2. Deploy the httpbin app in the apps-2 namespace by running the following command:

      $ oc apply -n apps-2 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/httpbin/httpbin.yaml

4. Deploy the sleep app in the namespaces:

   1. Deploy the sleep app in the apps-1 namespace by running the following command:

      $ oc apply -n apps-1 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/sleep/sleep.yaml

   2. Deploy the sleep app in the apps-2 namespace by running the following command:

      $ oc apply -n apps-2 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/sleep/sleep.yaml

5. Verify that the created apps have sidecars injected:

   1. Verify that the created apps have sidecars injected for apps-1 namespace by running the following command:

      $ oc get pods -n apps-1

   2. Verify that the created apps have sidecars injected for apps-2 namespace by running the following command:

      $ oc get pods -n apps-2

6. Create a mesh-wide strict mutual Transport Layer Security (mTLS) policy similar to the following example:

   Note

   Enabling PeerAuthentication in strict mTLS mode verifies correct certificate distribution and functional mTLS communication between workloads.

   apiVersion: security.istio.io/v1beta1
   kind: PeerAuthentication
   metadata:
     name: default
     namespace: istio-system
   spec:
     mtls:
       mode: STRICT

7. Apply the mTLS policy by running the following command:

   $ oc apply -f peer_auth.yaml

8. Verify that the apps-1/sleep app can access the apps-2/httpbin service by running the following command:

   $ oc -n apps-1 exec "$(oc -n apps-1 get pod \
     -l app=sleep -o jsonpath={.items..metadata.name})" \
     -c sleep -- curl -sIL http://httpbin.apps-2.svc.cluster.local:8000

   You should see output similar to the following example:

   HTTP/1.1 200 OK
   access-control-allow-credentials: true
   access-control-allow-origin: *
   content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' camo.githubusercontent.com
   content-type: text/html; charset=utf-8
   date: Wed, 18 Jun 2025 09:20:55 GMT
   x-envoy-upstream-service-time: 14
   server: envoy
   transfer-encoding: chunked

9. Verify that the apps-2/sleep app can access the apps-1/httpbin service by running the following command:

   $ oc -n apps-2 exec "$(oc -n apps-1 get pod \
     -l app=sleep -o jsonpath={.items..metadata.name})" \
     -c sleep -- curl -sIL http://httpbin.apps-2.svc.cluster.local:8000

   You should see output similar to the following example:

   HTTP/1.1 200 OK
   access-control-allow-credentials: true
   access-control-allow-origin: *
   content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' camo.githubusercontent.com
   content-type: text/html; charset=utf-8
   date: Wed, 18 Jun 2025 09:21:23 GMT
   x-envoy-upstream-service-time: 16
   server: envoy
   transfer-encoding: chunked

10. Verify that the httpbin workload certificate matches as expected by running the following command:

    $ istioctl proxy-config secret -n apps-1 \
      $(oc get pods -n apps-1 -o jsonpath='{.items..metadata.name}' --selector app=httpbin) \
      -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' \
      | base64 --decode | openssl x509 -text -noout

    You should see output similar to the following example:

    ...
    Issuer: O = cert-manager + O = cluster.local, CN = istio-ca
    ...
    X509v3 Subject Alternative Name:
    URI:spiffe://cluster.local/ns/apps-1/sa/httpbin

Procedure

1. Create the following namespaces:

   1. Create the apps-1 namespace by running the following command:

      $ oc new-project apps-1

   2. Create the apps-2 namespace by running the following command:

      $ oc new-project apps-2

2. Add the istio-discovered=enabled label on the namespaces:

   1. Add the istio-discovered=enabled label on the apps-1 namespace by running the following command:

      $ oc label namespaces apps-1 istio-discovery=enabled

   2. Add the istio-discovered=enabled label on the apps-2 namespace by running the following command:

      $ oc label namespaces apps-2 istio-discovery=enabled

3. Deploy the httpbin app in the namespaces:

   1. Deploy the httpbin app in the apps-1 namespace by running the following command:

      $ oc apply -n apps-1 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/httpbin/httpbin.yaml

   2. Deploy the httpbin app in the apps-2 namespace by running the following command:

      $ oc apply -n apps-2 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/httpbin/httpbin.yaml

4. Deploy the sleep app in the namespaces:

   1. Deploy the sleep app in the apps-1 namespace by running the following command:

      $ oc apply -n apps-1 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/sleep/sleep.yaml

   2. Deploy the sleep app in the apps-2 namespace by running the following command:

      $ oc apply -n apps-2 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/sleep/sleep.yaml

5. Add the apps-1 and apps-2 applications to the mesh in ambient mode:

   1. Apply the istio.io/dataplane-mode=ambient label to the apps-1 namespace as follows:

      $ oc label namespace apps-1 `istio.io/dataplane-mode=ambient`

   2. Apply the istio.io/dataplane-mode=ambient label to the apps-2 namespace as follows:

      $ oc label namespace apps-2 `istio.io/dataplane-mode=ambient`

      Note

      You can also add the label to individual pods to include only individual workloads.

6. Confirm that ztunnel successfully opened listening sockets inside the pod network namespace, as shown in the following example:

   $ istioctl ztunnel-config workloads --namespace ztunnel

   You will get an output similar to the following example:

   # NAMESPACE    POD NAME                 ADDRESS     NODE                               WAYPOINT PROTOCOL
   apps-1       httpbin-5c6c796d88-mt864 10.131.0.28 user-rhos-d-2-7qxgk-worker-0-9s7rf None     HBONE
   apps-1       sleep-fb964478b-qq4cs    10.131.0.30 user-rhos-d-2-7qxgk-worker-0-9s7rf None     HBONE
   apps-2       httpbin-5c6c796d88-jvt52 10.131.0.29 user-rhos-d-2-7qxgk-worker-0-9s7rf None     HBONE
   apps-2       sleep-fb964478b-xr9qw    10.131.0.31 user-rhos-d-2-7qxgk-worker-0-9s7rf None     HBONE
   istio-cni    istio-cni-node-4xvkv     10.128.2.79 user-rhos-d-2-7qxgk-worker-0-jztd5 None     TCP
   istio-cni    istio-cni-node-czhkx     10.129.1.40 user-rhos-d-2-7qxgk-master-1       None     TCP
   istio-cni    istio-cni-node-kpnmr     10.128.0.18 user-rhos-d-2-7qxgk-master-0       None     TCP
   istio-cni    istio-cni-node-nd88z     10.130.0.72 user-rhos-d-2-7qxgk-master-2       None     TCP
   istio-cni    istio-cni-node-tmx5l     10.131.0.26 user-rhos-d-2-7qxgk-worker-0-9s7rf None     TCP
   istio-system istiod-58f9796ddd-pdnh2  10.128.2.77 user-rhos-d-2-7qxgk-worker-0-jztd5 None     TCP
   ztunnel      ztunnel-64vqw            10.129.1.39 user-rhos-d-2-7qxgk-master-1       None     TCP
   ztunnel      ztunnel-bnbgh            10.128.2.78 user-rhos-d-2-7qxgk-worker-0-jztd5 None     TCP
   ztunnel      ztunnel-jkzbl            10.130.0.71 user-rhos-d-2-7qxgk-master-2       None     TCP
   ztunnel      ztunnel-p6pdf            10.128.0.17 user-rhos-d-2-7qxgk-master-0       None     TCP
   ztunnel      ztunnel-xbpx2            10.131.0.25 user-rhos-d-2-7qxgk-worker-0-9s7rf None     TCP

7. Create a mesh-wide strict mutual Transport Layer Security (mTLS) policy similar to the following example:

   Note

   Enabling PeerAuthentication in strict mTLS mode verifies correct certificate distribution and functional mTLS communication between workloads.

   apiVersion: security.istio.io/v1beta1
   kind: PeerAuthentication
   metadata:
     name: default
     namespace: istio-system
   spec:
     mtls:
       mode: STRICT

8. Apply the mTLS policy by running the following command:

   $ oc apply -f peer_auth.yaml

9. Verify that the apps-1/sleep app can access the apps-2/httpbin service by running the following command:

   $ oc -n apps-1 exec "$(oc -n apps-1 get pod \
     -l app=sleep -o jsonpath={.items..metadata.name})" \
     -c sleep -- curl -sIL http://httpbin.apps-2.svc.cluster.local:8000

   You will get an output similar to the following example:

   HTTP/1.1 200 OK
   access-control-allow-credentials: true
   access-control-allow-origin: *
   content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' camo.githubusercontent.com
   content-type: text/html; charset=utf-8
   date: Wed, 18 Jun 2025 09:20:55 GMT
   x-envoy-upstream-service-time: 14
   server: envoy
   transfer-encoding: chunked

10. Verify that the apps-2/sleep app can access the apps-1/httpbin service by running the following command:

    $ oc -n apps-2 exec "$(oc -n apps-1 get pod \
      -l app=sleep -o jsonpath={.items..metadata.name})" \
      -c sleep -- curl -sIL http://httpbin.apps-2.svc.cluster.local:8000

    You will get an output similar to the following example:

    HTTP/1.1 200 OK
    access-control-allow-credentials: true
    access-control-allow-origin: *
    content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' camo.githubusercontent.com
    content-type: text/html; charset=utf-8
    date: Wed, 18 Jun 2025 09:21:23 GMT
    x-envoy-upstream-service-time: 16
    server: envoy
    transfer-encoding: chunked

11. Verify that the httpbin workload certificate matches as expected by running the following command:

    $ istioctl proxy-config secret -n apps-1 \
      $(oc get pods -n apps-1 -o jsonpath='{.items..metadata.name}' --selector app=httpbin) \
      -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' \
      | base64 --decode | openssl x509 -text -noout

    You will get an output similar to the following example:

    ...
    Issuer: O = cert-manager + O = cluster.local, CN = istio-ca
    ...
    X509v3 Subject Alternative Name:
    URI:spiffe://cluster.local/ns/apps-1/sa/httpbin

Procedure

1. Remove the IstioCSR custom resource by running the following command:

   $ oc -n <istio-csr_project_name> delete istiocsrs.operator.openshift.io default

2. Remove the related resources:

   1. List the cluster scoped-resources by running the following command:

      $ oc get clusterrolebindings,clusterroles -l "app=cert-manager-istio-csr,app.kubernetes.io/name=cert-manager-istio-csr"

      Save the names of the listed resources for later reference.

   2. List the resources in istio-csr agent deployed namespace by running the following command:

      $ oc get certificate,deployments,services,serviceaccounts -l "app=cert-manager-istio-csr,app.kubernetes.io/name=cert-manager-istio-csr" -n <istio_csr_project_name>

      Save the names of the listed resources for later reference.

   3. List the resources in Red Hat OpenShift Service Mesh or Istio deployed namespaces by running the following command:

      $ oc get roles,rolebindings \
        -l "app=cert-manager-istio-csr,app.kubernetes.io/name=cert-manager-istio-csr" \
        -n <istio_csr_project_name>

      Save the names of the listed resources for later reference.

   4. For each resource listed in previous steps, delete the resources by running the following command:

      $ oc -n <istio_csr_project_name> delete <resource_type>/<resource_name>