Chapter 5. OpenShift Service Mesh and cert-manager

Procedure

1. Create the istio-system namespace by running the following command:

   $ oc create namespace istio-system

   Copy to Clipboard Toggle word wrap

2. Patch the cert-manager Operator to install the istio-csr agent by running the following command:

   $ oc -n cert-manager-operator patch subscription openshift-cert-manager-operator \
     --type='merge' -p \
     '{"spec":{"config":{"env":[{"name":"UNSUPPORTED_ADDON_FEATURES","value":"IstioCSR=true"}]}}}'

   Copy to Clipboard Toggle word wrap

3. Create the root certificate authority (CA) issuer by creating an Issuer object for the istio-csr agent:

   1. Create a new project for installing the istio-csr agent by running the following command:

      $ oc new-project istio-csr

      Copy to Clipboard Toggle word wrap

   2. Create an Issuer object similar to the following example:

      Note

      The selfSigned issuer is intended for demonstration, testing, or proof-of-concept environments. For production deployments, use a secure and trusted CA.

      Example issuer.yaml file

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: selfsigned
        namespace: istio-system
      spec:
        selfSigned: {}
      ---
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: istio-ca
        namespace: istio-system
      spec:
        isCA: true
        duration: 87600h
        secretName: istio-ca
        commonName: istio-ca
        privateKey:
          algorithm: ECDSA
          size: 256
        subject:
          organizations:
            - cluster.local
            - cert-manager
        issuerRef:
          name: selfsigned
          kind: Issuer
          group: cert-manager.io
      ---
      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: istio-ca
        namespace: istio-system
      spec:
        ca:
          secretName: istio-ca

      Copy to Clipboard Toggle word wrap

   3. Create the objects by running the following command:

      $ oc apply -f issuer.yaml

      Copy to Clipboard Toggle word wrap

   4. Wait for the istio-ca certificate to contain the "Ready" status condition by running the following command:

      $ oc wait --for=condition=Ready certificates/istio-ca -n istio-system

      Copy to Clipboard Toggle word wrap

4. Create the IstioCSR custom resource:

   1. Create the IstioCSR custom resource similar to the following example:

      Example istioCSR.yaml file

      apiVersion: operator.openshift.io/v1alpha1
      kind: IstioCSR
      metadata:
        name: default
        namespace: istio-csr
      spec:
        istioCSRConfig:
          certManager:
            issuerRef:
              name: istio-ca
              kind: Issuer
              group: cert-manager.io
          istiodTLSConfig:
            trustDomain: cluster.local
          istio:
            namespace: istio-system

      Copy to Clipboard Toggle word wrap

   2. Create the istio-csr agent by by running the following command:

      $ oc create -f istioCSR.yaml

      Copy to Clipboard Toggle word wrap

   3. Verify that the istio-csr deployment is ready by running the following command:

      $ oc get deployment -n istio-csr

      Copy to Clipboard Toggle word wrap

5. Install the istio resource:

   Note

   The configuration disables the built-in CA server for Istio and forwards certificate signing requests from istiod to the istio-csr agent. The istio-csr agent obtains certificates for both istiod and mesh workloads from the cert-manager Operator. The istiod TLS certificate that is generated by the istio-csr agent is mounted into the pod at a known location for use.

   1. Create the Istio object similar to the following example:

      Example istio.yaml file

      apiVersion: sailoperator.io/v1
      kind: Istio
      metadata:
        name: default
      spec:
        version: v1.24-latest
        namespace: istio-system
        values:
          global:
            caAddress: cert-manager-istio-csr.istio-csr.svc:443
          pilot:
            env:
              ENABLE_CA_SERVER: "false"

      Copy to Clipboard Toggle word wrap

   2. Create the Istio resource by running the following command:

      $ oc apply -f istio.yaml

      Copy to Clipboard Toggle word wrap

   3. Verify that the istio resource displays the "Ready" status condition by running the following command:

      $ oc wait --for=condition=Ready istios/default -n istio-system

      Copy to Clipboard Toggle word wrap

1. Create the namespaces:

   1. Create the apps-1 namespace by running the following command:

      $ oc new-project apps-1

      Copy to Clipboard Toggle word wrap

   2. Create the apps-2 namespace by running the following command:

      $ oc new-project apps-2

      Copy to Clipboard Toggle word wrap

2. Add the istio-injection=enabled label on the namespaces:

   1. Add the istio-injection=enabled label on the apps-1 namespace by running the following command:

      $ oc label namespaces apps-1 istio-injection=enabled

      Copy to Clipboard Toggle word wrap

   2. Add the istio-injection=enabled label on the apps-2 namespace by running the following command:

      $ oc label namespaces apps-2 istio-injection=enabled

      Copy to Clipboard Toggle word wrap

3. Deploy the httpbin app in the namespaces:

   1. Deploy the httpbin app in the apps-1 namespace by running the following command:

      $ oc apply -n apps-1 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/httpbin/httpbin.yaml

      Copy to Clipboard Toggle word wrap

   2. Deploy the httpbin app in the apps-2 namespace by running the following command:

      $ oc apply -n apps-2 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/httpbin/httpbin.yaml

      Copy to Clipboard Toggle word wrap

4. Deploy the sleep app in the namespaces:

   1. Deploy the sleep app in the apps-1 namespace by running the following command:

      $ oc apply -n apps-1 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/sleep/sleep.yaml

      Copy to Clipboard Toggle word wrap

   2. Deploy the sleep app in the apps-2 namespace by running the following command:

      $ oc apply -n apps-2 -f https://raw.githubusercontent.com/openshift-service-mesh/istio/release-1.24/samples/sleep/sleep.yaml

      Copy to Clipboard Toggle word wrap

5. Verify that the created apps have sidecars injected:

   1. Verify that the created apps have sidecars injected for apps-1 namespace by running the following command:

      $ oc get pods -n apps-1

      Copy to Clipboard Toggle word wrap

   2. Verify that the created apps have sidecars injected for apps-2 namespace by running the following command:

      $ oc get pods -n apps-2

      Copy to Clipboard Toggle word wrap

6. Create a mesh-wide strict mutual Transport Layer Security (mTLS) policy similar to the following example:

   Note

   Enabling PeerAuthentication in strict mTLS mode verifies that certificates are distributed correctly and that mTLS communication functions between workloads.

   Example peer_auth.yaml file

   apiVersion: security.istio.io/v1beta1
   kind: PeerAuthentication
   metadata:
     name: default
     namespace: istio-system
   spec:
     mtls:
       mode: STRICT

   Copy to Clipboard Toggle word wrap

7. Apply the mTLS policy by running the following command:

   $ oc apply -f peer_auth.yaml

   Copy to Clipboard Toggle word wrap

8. Verify that the apps-1/sleep app can access the apps-2/httpbin service by running the following command:

   $ oc -n apps-1 exec "$(oc -n apps-1 get pod \
     -l app=sleep -o jsonpath={.items..metadata.name})" \
     -c sleep -- curl -sIL http://httpbin.apps-2.svc.cluster.local:8000

   Copy to Clipboard Toggle word wrap

   Example output

   HTTP/1.1 200 OK
   access-control-allow-credentials: true
   access-control-allow-origin: *
   content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' camo.githubusercontent.com
   content-type: text/html; charset=utf-8
   date: Wed, 18 Jun 2025 09:20:55 GMT
   x-envoy-upstream-service-time: 14
   server: envoy
   transfer-encoding: chunked

   Copy to Clipboard Toggle word wrap

9. Verify that the apps-2/sleep app can access the apps-1/httpbin service by running the following command:

   $ oc -n apps-2 exec "$(oc -n apps-1 get pod \
     -l app=sleep -o jsonpath={.items..metadata.name})" \
     -c sleep -- curl -sIL http://httpbin.apps-2.svc.cluster.local:8000

   Copy to Clipboard Toggle word wrap

   Example output

   HTTP/1.1 200 OK
   access-control-allow-credentials: true
   access-control-allow-origin: *
   content-security-policy: default-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' camo.githubusercontent.com
   content-type: text/html; charset=utf-8
   date: Wed, 18 Jun 2025 09:21:23 GMT
   x-envoy-upstream-service-time: 16
   server: envoy
   transfer-encoding: chunked

   Copy to Clipboard Toggle word wrap

10. Verify that the httpbin workload certificate matches as expected by running the following command:

    $ istioctl proxy-config secret -n apps-1 \
      $(oc get pods -n apps-1 -o jsonpath='{.items..metadata.name}' --selector app=httpbin) \
      -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' \
      | base64 --decode | openssl x509 -text -noout

    Copy to Clipboard Toggle word wrap

    Example output

    ...
    Issuer: O = cert-manager + O = cluster.local, CN = istio-ca
    ...
    X509v3 Subject Alternative Name:
    URI:spiffe://cluster.local/ns/apps-1/sa/httpbin

    Copy to Clipboard Toggle word wrap

Procedure

1. Remove the IstioCSR custom resource by running the following command:

   $ oc -n <istio-csr_project_name> delete istiocsrs.operator.openshift.io default

   Copy to Clipboard Toggle word wrap

2. Remove the related resources:

   1. List the cluster scoped-resources by running the following command:

      $ oc get clusterrolebindings,clusterroles -l "app=cert-manager-istio-csr,app.kubernetes.io/name=cert-manager-istio-csr"

      Copy to Clipboard Toggle word wrap

      Save the names of the listed resources for later reference.

   2. List the resources in istio-csr agent deployed namespace by running the following command:

      $ oc get certificate,deployments,services,serviceaccounts -l "app=cert-manager-istio-csr,app.kubernetes.io/name=cert-manager-istio-csr" -n <istio_csr_project_name>

      Copy to Clipboard Toggle word wrap

      Save the names of the listed resources for later reference.

   3. List the resources in Red Hat OpenShift Service Mesh or Istio deployed namespaces by running the following command:

      $ oc get roles,rolebindings \
        -l "app=cert-manager-istio-csr,app.kubernetes.io/name=cert-manager-istio-csr" \
        -n <istio_csr_project_name>

      Copy to Clipboard Toggle word wrap

      Save the names of the listed resources for later reference.

   4. For each resource listed in previous steps, delete the resources by running the following command:

      $ oc -n <istio_csr_project_name> delete <resource_type>/<resource_name>

      Copy to Clipboard Toggle word wrap