Chapter 5. OpenShift Service Mesh and cert-manager

Procedure

1. Create the istio-system namespace by running the following command:

   $ oc create namespace istio-system

2. Patch the cert-manager Operator to install the istio-csr agent by running the following command:

   $ oc -n cert-manager-operator patch subscription openshift-cert-manager-operator \
     --type='merge' -p \
     '{"spec":{"config":{"env":[{"name":"UNSUPPORTED_ADDON_FEATURES","value":"IstioCSR=true"}]}}}'

3. Create the root certificate authority (CA) issuer by creating an Issuer object for the istio-csr agent:

   1. Create a new project for installing the istio-csr agent by running the following command:

      $ oc new-project istio-csr

   2. Create an Issuer object similar to the following example:

      Note

      The selfSigned issuer is intended for demonstration, testing, or proof-of-concept environments. For production deployments, use a secure and trusted CA.

      Example issuer.yaml file

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: selfsigned
        namespace: istio-system
      spec:
        selfSigned: {}
      ---
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: istio-ca
        namespace: istio-system
      spec:
        isCA: true
        duration: 87600h
        secretName: istio-ca
        commonName: istio-ca
        privateKey:
          algorithm: ECDSA
          size: 256
        subject:
          organizations:
            - cluster.local
            - cert-manager
        issuerRef:
          name: selfsigned
          kind: Issuer
          group: cert-manager.io
      ---
      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: istio-ca
        namespace: istio-system
      spec:
        ca:
          secretName: istio-ca

   3. Create the objects by running the following command:

      $ oc apply -f issuer.yaml

   4. Wait for the istio-ca certificate to contain the "Ready" status condition by running the following command:

      $ oc wait --for=condition=Ready certificates/istio-ca -n istio-system

4. Create the IstioCSR custom resource:

   1. Create the IstioCSR custom resource similar to the following example:

      Example istioCSR.yaml file

      apiVersion: operator.openshift.io/v1alpha1
      kind: IstioCSR
      metadata:
        name: default
        namespace: istio-csr
      spec:
        istioCSRConfig:
          certManager:
            issuerRef:
              name: istio-ca
              kind: Issuer
              group: cert-manager.io
          istiodTLSConfig:
            trustDomain: cluster.local
          istio:
            namespace: istio-system

   2. Create the istio-csr agent by by running the following command:

      $ oc create -f istioCSR.yaml

   3. Verify that the istio-csr deployment is ready by running the following command:

      $ oc get deployment -n istio-csr

5. Install the istio resource:

   Note

   The configuration disables the built-in CA server for Istio and forwards certificate signing requests from istiod to the istio-csr agent. The istio-csr agent obtains certificates for both istiod and mesh workloads from the cert-manager Operator. The istiod TLS certificate that is generated by the istio-csr agent is mounted into the pod at a known location for use.

   1. Create the Istio object similar to the following example:

      Example istio.yaml file

      apiVersion: sailoperator.io/v1
      kind: Istio
      metadata:
        name: default
      spec:
        version: v1.24-latest
        namespace: istio-system
        values:
          global:
            caAddress: cert-manager-istio-csr.istio-csr.svc:443
          pilot:
            env:
              ENABLE_CA_SERVER: "false"

   2. Create the Istio resource by running the following command:

      $ oc apply -f istio.yaml

   3. Verify that the istio resource displays the "Ready" status condition by running the following command:

      $ oc wait --for=condition=Ready istios/default -n istio-system

1. Create the namespaces:

   1. Create the apps-1 namespace by running the following command:

      $ oc new-project apps-1

   2. Create the apps-2 namespace by running the following command:

      $ oc new-project apps-2

2. Add the istio-injection=enabled label on the namespaces:

   1. Add the istio-injection=enabled label on the apps-1 namespace by running the following command:

      $ oc label namespaces apps-1 istio-injection=enabled

   2. Add the istio-injection=enabled label on the apps-2 namespace by running the following command:

      $ oc label namespaces apps-2 istio-injection=enabled

3. Deploy the httpbin app in the namespaces:

   1. Deploy the httpbin app in the apps-1 namespace by running the following command:

      $ oc apply -n apps-1 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/httpbin/httpbin.yaml

   2. Deploy the httpbin app in the apps-2 namespace by running the following command:

      $ oc apply -n apps-2 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/httpbin/httpbin.yaml

4. Deploy the sleep app in the namespaces:

   1. Deploy the sleep app in the apps-1 namespace by running the following command:

      $ oc apply -n apps-1 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/sleep/sleep.yaml

   2. Deploy the sleep app in the apps-2 namespace by running the following command:

      $ oc apply -n apps-2 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/sleep/sleep.yaml

5. Verify that the created apps have sidecars injected:

   1. Verify that the created apps have sidecars injected for apps-1 namespace by running the following command:

      $ oc get pods -n apps-1

   2. Verify that the created apps have sidecars injected for apps-2 namespace by running the following command:

      $ oc get pods -n apps-2

6. Create a mesh-wide strict mutual Transport Layer Security (mTLS) policy similar to the following example:

   Note

   Enabling PeerAuthentication in strict mTLS mode verifies that certificates are distributed correctly and that mTLS communication functions between workloads.

   Example peer_auth.yaml file

   apiVersion: security.istio.io/v1beta1
   kind: PeerAuthentication
   metadata:
     name: default
     namespace: istio-system
   spec:
     mtls:
       mode: STRICT

7. Apply the mTLS policy by running the following command:

   $ oc apply -f peer_auth.yaml

8. Verify that the apps-1/sleep app can access the apps-2/httpbin service by running the following command:

   $ oc -n apps-1 exec "$(oc -n apps-1 get pod \
     -l app=sleep -o jsonpath={.items..metadata.name})" \
     -c sleep -- curl -sIL http://httpbin.apps-2.svc.cluster.local:8000

   Example output

   HTTP/1.1 200 OK
   access-control-allow-credentials: true
   access-control-allow-origin: *
   content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' camo.githubusercontent.com
   content-type: text/html; charset=utf-8
   date: Wed, 18 Jun 2025 09:20:55 GMT
   x-envoy-upstream-service-time: 14
   server: envoy
   transfer-encoding: chunked

9. Verify that the apps-2/sleep app can access the apps-1/httpbin service by running the following command:

   $ oc -n apps-2 exec "$(oc -n apps-1 get pod \
     -l app=sleep -o jsonpath={.items..metadata.name})" \
     -c sleep -- curl -sIL http://httpbin.apps-2.svc.cluster.local:8000

   Example output

   HTTP/1.1 200 OK
   access-control-allow-credentials: true
   access-control-allow-origin: *
   content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' camo.githubusercontent.com
   content-type: text/html; charset=utf-8
   date: Wed, 18 Jun 2025 09:21:23 GMT
   x-envoy-upstream-service-time: 16
   server: envoy
   transfer-encoding: chunked

10. Verify that the httpbin workload certificate matches as expected by running the following command:

    $ istioctl proxy-config secret -n apps-1 \
      $(oc get pods -n apps-1 -o jsonpath='{.items..metadata.name}' --selector app=httpbin) \
      -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' \
      | base64 --decode | openssl x509 -text -noout

    Example output

    ...
    Issuer: O = cert-manager + O = cluster.local, CN = istio-ca
    ...
    X509v3 Subject Alternative Name:
    URI:spiffe://cluster.local/ns/apps-1/sa/httpbin

Procedure

1. Remove the IstioCSR custom resource by running the following command:

   $ oc -n <istio-csr_project_name> delete istiocsrs.operator.openshift.io default

2. Remove the related resources:

   1. List the cluster scoped-resources by running the following command:

      $ oc get clusterrolebindings,clusterroles -l "app=cert-manager-istio-csr,app.kubernetes.io/name=cert-manager-istio-csr"

      Save the names of the listed resources for later reference.

   2. List the resources in istio-csr agent deployed namespace by running the following command:

      $ oc get certificate,deployments,services,serviceaccounts -l "app=cert-manager-istio-csr,app.kubernetes.io/name=cert-manager-istio-csr" -n <istio_csr_project_name>

      Save the names of the listed resources for later reference.

   3. List the resources in Red Hat OpenShift Service Mesh or Istio deployed namespaces by running the following command:

      $ oc get roles,rolebindings \
        -l "app=cert-manager-istio-csr,app.kubernetes.io/name=cert-manager-istio-csr" \
        -n <istio_csr_project_name>

      Save the names of the listed resources for later reference.

   4. For each resource listed in previous steps, delete the resources by running the following command:

      $ oc -n <istio_csr_project_name> delete <resource_type>/<resource_name>