Chapter 10. Zero Trust Workload Identity Manager

1. The SPIRE, SPIRE Agent, SPIFFE CSI Driver, and the SPIRE OIDC Discovery Provider operands are deployed and managed by Zero Trust Workload Identity Manager via associated customer resource definitions (CRDs).
2. Watches are then registered for relevant Kubernetes resources and the necessary SPIRE CRDs are applied to the cluster.
3. The CR for the ZeroTrustWorkloadIdentityManager resource named cluster is deployed and managed by a controller.

4. To deploy the SPIRE Server, SPIRE Agent, SPIFFE CSI Driver, and SPIRE OIDC Discovery Provider, you need to create a custom resource of a each certain type and name it cluster. The custom resource types are as follows:

   • SPIRE Server - SpireServer
   • SPIRE Agent - SpireAgent
   • SPIFFE CSI Driver - SpiffeCSIDriver
   • SPIRE OIDC discovery provider - SpireOIDCDiscoveryProvider
5. When a node starts, the SPIRE Agent initializes, and connects to the SPIRE Server.
6. The SPIRE Agent begins the node attestation process. The agent collects information on the node’s identity such as label name and namespace. The agent securely provides the information it gathered through the attestation to the SPIRE Server.
7. The SPIRE Server then evaluates this information against its configured attestation policies and registration entries. If successful, the server generates an agent SVID and the Trust Bundle (CA Certificate) and securely sends this back to the SPIRE Agent.
8. A workload starts on the node and needs a secure identity. The workload connects to the agent’s Workload API and requests a SVID.
9. The SPIRE Agent receives the request and begins a workload attestation to gather information about the workload.
10. After the SPIRE Agent gathers the information, the information is sent to the SPIRE Server and the server checks its configured registration entries.
11. The SPIRE Agent receives the workload SVID and Trust Bundle and passes it on to the workload. The workload can now present their SVIDs to other SPIFFE-aware devices to communicate with them.

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Go to Ecosystem Software Catalog.
3. Search for Zero Trust Workload Identity Manager.

4. On the Install Operator page:

   1. Update the Update channel, if necessary. The channel defaults to stable-v1, which installs the latest stable-v1 release of the Zero Trust Workload Identity Manager.

   2. Choose the Installed Namespace for the Operator. The default Operator namespace is zero-trust-workload-identity-manager.

      If the zero-trust-workload-identity-manager namespace does not exist, it is created for you.

      Note

      The Operator and operands are deployed in the same namespace.

   3. Select an Update Approval strategy

      • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
      • The Manual strategy requires a user with appropriate credentials to approve the Operator update.
5. Click Install.

Verification

1. Navigate to Ecosystem Installed Operators.

   1. Verify that Zero Trust Workload Identity Manager is listed with a Status of Succeeded in the zero-trust-workload-identity-manager namespace.

   2. Verify that Zero Trust Workload Identity Manager controller manager deployment is ready and available by running the following command:

      $ oc get deployment -l name=zero-trust-workload-identity-manager -n zero-trust-workload-identity-manager

      Example output

      NAME                                                           READY UP-TO-DATE AVAILABLE AGE
      zero-trust-workload-identity-manager-controller-manager-6c4djb 1/1   1          1         43m

2. To check the Operator logs, run the following command:

   $ oc logs -f deployment/zero-trust-workload-identity-manager -n zero-trust-workload-identity-manager

Procedure

1. Create a new project named zero-trust-workload-identity-manager by running the following command:

   $ oc new-project zero-trust-workload-identity-manager

2. Create an OperatorGroup object:

   1. Create a YAML file, for example, operatorGroup.yaml, with the following content:

      Example operatorGroup.yaml

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-zero-trust-workload-identity-manager
        namespace: zero-trust-workload-identity-manager
      spec:
        upgradeStrategy: Default

   2. Create the OperatorGroup object by running the following command:

      $ oc create -f operatorGroup.yaml

3. Create a Subscription object:

   1. Create a YAML file, for example, subscription.yaml, that defines the Subscription object:

      Example subscription.yaml

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: openshift-zero-trust-workload-identity-manager
        namespace: zero-trust-workload-identity-manager
      spec:
        channel: stable-v1
        name: openshift-zero-trust-workload-identity-manager
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        installPlanApproval: Automatic

   2. Create the Subscription object by running the following command:

      $ oc create -f subscription.yaml

Procedure

1. Create the SpireServer CR:

   1. Create a YAML file that defines the SpireServer CR, for example, SpireServer.yaml:

      Example SpireServer.yaml

      apiVersion: operator.openshift.io/v1alpha1
      kind: SpireServer
      metadata:
       name: cluster
      spec:
        logLevel: "info"
        logFormat: "text"
        jwtIssuer: "https://oidc-discovery.apps.cluster.example.com"
        caValidity: "24h"
        defaultX509Validity: "1h"
        defaultJWTValidity: "5m"
        jwtKeyType: "rsa-2048"
        caSubject:
          country: "US"
          organization: "Example Corporation"
          commonName: "SPIRE Server CA"
        persistence:
          size: "5Gi"
          accessMode: "ReadWriteOnce"
          storageClass: "gp3-csi"
        datastore:
          databaseType: "sqlite3"
          connectionString: "/run/spire/data/datastore.sqlite3"
          tlsSecretName: ""
          maxOpenConns: 100
          maxIdleConns: 10
          connMaxLifetime: 0
          disableMigration: "false"

      where:

      metadata.name

      Specifies that the value must be cluster.

      spec.logLevel

      Specifies the logging level for the SPIRE Server. The valid options are debug, info, warn, and error.

      spec.logFormat

      Specifies the logging format for the SPIRE Server. The valid options are text and json.

      spec.jwtIssuer

      Specifies the JWT issuer URL. Must be a valid HTTPS or HTTP URL with a maximum length of 512 characters.

      spec.caValidity

      Specifies the validity period (Time to Live (TTL)) for the SPIRE Server’s CA certificate. This determines how long the server’s root or intermediate certificate is valid. The format is a duration string (for example, 24h, 168h).

      spec.defaultX509Validity

      Specifies the default validity period (TTL) for X.509 SVIDs issued to workloads. This value is used if a specific TTL is not configured for a registration entry.

      spec.defaultJWTValidity

      Specifies thedefault validity period (TTL) for JWT SVIDs issued to workloads. This value is used if a specific TTL is not configured for a registration entry.

      spec.wtKeyType

      Specifies the key type used for JWT signing. The valid options are rsa-2048, rsa-4096, ec-p256, and ec-p384. This field is optional.

      spec.caSubject.country

      Specifies the country for the SPIRE Server certificate authority (CA). Must be an ISO 3166-1 alpha-2 country code (2 characters).

      spec.caSubject.organization

      Specifies the organization for the SPIRE Server CA. Maximum length is 64 characters.

      spec.caSubject.commonName

      Specifies the common name for the SPIRE Server CA. Maximum length is 255 characters.

      spec.persistence.size

      Specifies the size of the persistent volume (for example, 1Gi, 5Gi). Once set, this field is immutable.

      spec.persistence.accessMode

      Specifies the access mode for the persistent volume. The valid options are ReadWriteOnce, ReadWriteOncePod, and ReadWriteMany. Once set, this field is immutable.

      spec.persistence.storageClass

      Specifies the storage class to be used for the PVC. Once set, this field is immutable.

      spec.datastore.databaseType

      Specifies the type of database to use for the datastore. The valid options are sql, sqlite3, postgres, mysql, aws_postgresql, and aws_mysql.

      spec.datastore.connectionString

      Specifies the connection string for the database. For PostgreSQL with SSL, include sslmode and certificate paths (for example, dbname=spire user=spire host=postgres.example.com sslmode=verify-full).

      spec.datastore.tlsSecretName

      Specifies the name of a Kubernetes Secret containing TLS certificates for database connections. The Secret will be mounted at /run/spire/db/certs. This field is optional.

      spec.datastore.maxOpenConns

      Specifies the maximum number of open database connections. Must be between 1 and 10000.

      spec.datastore.maxIdleConns

      Specifies the maximum number of idle database connections in the pool. Must be between 0 and 10000.

      spec.datastore.connMaxLifetime

      Specifies the maximum lifetime of a database connection in seconds. A value of 0 means connections are not closed due to age.

      spec.datastore.disableMigration

      Specifies whether to disable automatic database migration. The valid options are true and false.

   2. Apply the configuration by running the following command:

      $ oc apply -f SpireServer.yaml

Procedure

1. Create the SpireAgent CR:

   1. Create a YAML file that defines the SpireAgent CR, for example, SpireAgent.yaml:

      Example SpireAgent.yaml

      apiVersion: operator.openshift.io/v1alpha1
      kind: SpireAgent
      metadata:
       name: cluster
      spec:
        socketPath: "/run/spire/agent-sockets"
        logLevel: "info"
        logFormat: "text"
        nodeAttestor:
          k8sPSATEnabled: "true"
        workloadAttestors:
          k8sEnabled: "true"
          workloadAttestorsVerification:
            type: "auto"
            hostCertBasePath: "/etc/kubernetes"
            hostCertFileName: "kubelet-ca.crt"
          disableContainerSelectors: "false"
          useNewContainerLocator: "true"

      where:

      metadata.name

      Specifies that the value must be cluster.

      spec.socketPath

      Specifies the directory on the host where the SPIRE agent socket is created. This directory is shared with the SPIFFE CSI driver via the hostPath volume. Must match the SpiffeCSIDriver.spec.agentSocketPath for workloads to access the socket. Must be an absolute path with a maximum length of 256 characters.

      spec.logLevel

      Specifies the logging level for the SPIRE Server. The valid options are debug, info, warn, and error.

      spec.logFormat

      Specifies the logging format for the SPIRE Server. The valid options are text and json.

      spec.nodeAttestor.k8sPSATEnabled

      Specifies whether Kubernetes Projected Service Account Token (PSAT) node attestation is enabled. When enabled, the SPIRE agent uses K8s PSATs to prove its identity to the SPIRE server during node attestation. The valid options are true and false.

      spec.workloadAttestors.k8sEnabled

      Specifies whether the Kubernetes workload attestor is enabled. When enabled, the SPIRE agent can verify workload identities using Kubernetes pod information and service account tokens. The valid options are true and false.

      spec.workloadAttestors.workloadAttestorsVerification.type

      Specifies the kubelet certificate verification mode. The valid options are auto, hostCert, and skip.

      spec.workloadAttestors.workloadAttestorsVerification.hostCertBasePath

      Specifies the directory containing the kubelet CA certificate. Required when type is hostCert. Optional when type is auto (defaults to /etc/kubernetes if not specified).

      spec.workloadAttestors.workloadAttestorsVerification.hostCertFileName

      Specifies the file name for the kubelet’s CA certificate. When combined with hostCertBasePath, forms the full path. Required when type is hostCert. Optional when type is auto. Defaults to kubelet-ca.crt if not specified.

      spec.workloadAttestors.disableContainerSelectors

      Specifies whether to disable container selectors in the Kubernetes workload attestor. Set to true if using holdApplicationUntilProxyStarts in Istio. The valid options are true and false.

      spec.workloadAttestors.useNewContainerLocator

      Specifies enabling the new container locator algorithm that has support for cgroups v2. The valid options are true and false.

   2. Apply the configuration by running the following command:

      $ oc apply -f SpireAgent.yaml

Procedure

1. Create the SpiffeCSIDriver CR:

   1. Create a YAML file that defines the SpiffeCSIDriver CR object, for example, SpiffeCSIDriver.yaml:

      Example SpiffeCSIDriver.yaml

      apiVersion: operator.openshift.io/v1alpha1
      kind: SpiffeCSIDriver
      metadata:
       name: cluster
      spec:
        agentSocketPath: "/run/spire/agent-sockets"
        pluginName: "csi.spiffe.io"

      where:

      metadata.name

      Specifies that the name must be cluster.

      spec.agentSocketPath

      Specifies the path to the directory containing the SPIRE agent’s Workload API socket. This directory is bind-mounted into workload containers by the CSI driver. The directory is shared between the SPIRE agent and CSI driver via a hostPath volume. Must be an absolute path with a maximum length of 256 characters. This value must match SpireAgent.spec.socketPath for workloads to access the socket.

      spec.pluginName

      Specifies the name of the CSI plugin. This sets the CSI driver name that is deployed to the cluster and used in VolumeMount configurations. Must match the driver name referenced in the workload pods. Must be a valid domain name format (for example, csi.spiffe.io) with a maximum length of 127 characters.

   2. Apply the configuration by running the following command:

      $ oc apply -f SpiffeCSIDriver.yaml

Procedure

1. Create the SpireOIDCDiscoveryProvider CR:

   1. Create a YAML file that defines the SpireOIDCDiscoveryProvider CR, for example, SpireOIDCDiscoveryProvider.yaml:

      Example SpireOIDCDiscoveryProvider.yaml

      aapiVersion: operator.openshift.io/v1alpha1
      kind: SpireOIDCDiscoveryProvider
      metadata:
       name: cluster
      spec:
        logLevel: "info"
        logFormat: "text"
        csiDriverName: "csi.spiffe.io"
        jwtIssuer: "https://oidc-discovery.apps.cluster.example.com"
        replicaCount: 1
        managedRoute: "true"
        externalSecretRef: ""

      where:

      metadata.name

      Specifies that the value must be cluster.

      spec.logLevel

      Specifies the logging level for the SPIRE Server. The valid options are debug, info, warn, and error.

      spec.logFormat

      Specifies the logging format for the SPIRE Server. The valid options are text and json.

      spec.csiDriverName

      Specifies the name of the CSI driver to use for mounting the Workload API socket. This must match the SpiffeCSIDriver.spec.pluginName value for the OIDC provider to access SPIFFE identities. Must be a valid DNS subdomain format (for example, csi.spiffe.io) with a maximum length of 127 characters.

      spec.jwtIssuer

      Specifies the JWT issuer URL. Must be a valid HTTPS or HTTP URL with a maximum length of 512 characters. This value must match the SpireServer.spec.jwtIssuer value.

      spec.replicaCount

      Specifies the number of replicas for the OIDC Discovery Provider deployment. Must be between 1 and 5.

      spec.managedRoute

      Specifies whether the Operator automatically creates an OpenShift route for the OIDC Discovery Provider endpoints. Set to true to have the Operator automatically create and maintain an OpenShift route for OIDC discovery endpoints (*.apps.). Set to false for administrators to manually configure routes or ingress.

      spec.ternalSecretRef

      Specifies a reference to an externally managed secret that contains the TLS certificate for the OIDC Discovery Provider route host. Must be a valid Kubernetes secret reference name with a maximum length of 253 characters. This field is optional.

   2. Apply the configuration by running the following command:

      $ oc apply -f SpireOIDCDiscoveryProvider.yaml

Verification

1. Verify that the deployment of OIDC Discovery Provider is ready and available by running the following command:

   $ oc get deployment -l app.kubernetes.io/name=spiffe-oidc-discovery-provider -n zero-trust-workload-identity-manager

   Example output

   NAME                                    READY  UP-TO-DATE  AVAILABLE  AGE
   spire-spiffe-oidc-discovery-provider    1/1    1           1          2m58s

2. Verify that the status of OIDC Discovery Provider pods is Running by running the following command:

   $ oc get po -l app.kubernetes.io/name=spiffe-oidc-discovery-provider -n zero-trust-workload-identity-manager

   Example output

   NAME                                                    READY   STATUS    RESTARTS   AGE
   spire-spiffe-oidc-discovery-provider-64586d599f-lcc94   2/2     Running   0          7m15s

Procedure

1. Create a config map in the zero-trust-workload-identity-manager namespace by running the following command:

   $ oc create configmap trusted-ca -n zero-trust-workload-identity-manager

2. Inject the CA bundle that is trusted by OpenShift Container Platform into the config map by running the following command:

   $ oc label cm trusted-ca config.openshift.io/inject-trusted-cabundle=true -n zero-trust-workload-identity-manager

3. Update the subscription for the Zero Trust Workload Identity Manager to use the config map by running the following command:

   $ oc -n zero-trust-workload-identity-manager patch subscription openshift-zero-trust-workload-identity-manager --type='merge' -p '{"spec":{"config":{"env":[{"name":"TRUSTED_CA_BUNDLE_CONFIGMAP","value":"trusted-ca"}]}}}'

Verification

1. Verify that the operands have finished rolling out by running the following command:

   $ oc rollout status deployment/zero-trust-workload-identity-manager-controller-manager -n zero-trust-workload-identity-manager && \
   $ oc rollout status statefulset/spireserver -n zero-trust-workload-identity-manager && \
   $ oc rollout status daemonset/spire-agent -n zero-trust-workload-identity-manager && \
   $ oc rollout status deployment/spire-spiffe-oidc-discovery-provider -n zero-trust-workload-identity-manager

   Example output

   deployment "zero-trust-workload-identity-manager-controller-manager" successfully rolled out
   statefulset "spire-server" successfully rolled out
   daemonset "spire-agent" successfully rolled out
   deployment "spire-spiffe-oidc-discovery-provider" successfully rolled out

2. Verify that the CA bundle was mounted as a volume by running the following command:

   $ oc get deployment zero-trust-workload-identity-manager -n zero-trust-workload-identity-manager -o=jsonpath={.spec.template.spec.'containers[0].volumeMounts'}

   $ oc get statefulset spire-server -n zero-trust-workload-identity-manager -o jsonpath='{.spec.template.spec.containers[*].volumeMounts[?(@.name=="trusted-ca-bundle")]}'

   $ oc get daemonset spire-agent -n zero-trust-workload-identity-manager -o jsonpath='{.spec.template.spec.containers[*].volumeMounts[?(@.name=="trusted-ca-bundle")]}'

   $ oc get daemonset spire-spiffe-csi-driver -n zero-trust-workload-identity-manager -o jsonpath='{.spec.template.spec.containers[*].volumeMounts[?(@.name=="trusted-ca-bundle")]}'

   Example output

   [{{"mountPath":"/etc/pki/ca-trust/extracted/pem","name":"trusted-ca-bundle","readOnly":true}}]

3. Verify that the source of the CA bundle is the trusted-ca config map by running the following command:

   $ oc get deployment zero-trust-workload-identity-manager -n zero-trust-workload-identity-manager -o=jsonpath={.spec.template.spec.volumes}

   $ oc get statefulset spire-server -n zero-trust-workload-identity-manager -o=jsonpath='{.spec.template.spec.volumes}' | jq '.[] | select(.name=="trusted-ca-bundle")'

   $ oc get daemonset spire-agent -n zero-trust-workload-identity-manager -o=jsonpath='{.spec.template.spec.volumes}' | jq '.[] | select(.name=="trusted-ca-bundle")'

   $ oc get deployment spire-spiffe-oidc-discovery-provider -n zero-trust-workload-identity-manager -o=jsonpath='{.spec.template.spec.volumes}' | jq '.[] | select(.name=="trusted-ca-bundle")'

   Example output

   {
     "configMap": {
       "defaultMode": 420,
       "items": [
         {
           "key": "ca-bundle.crt",
           "path": "tls-ca-bundle.pem"
         }
       ],
       "name": "trusted-ca"
     },
     "name": "trusted-ca-bundle"
   }

Procedure

1. Configure the SpireServer custom resource on each cluster to enable federation with the https_spiffe profile. The https_spiffe profile uses SPIFFE-based TLS authentication, where SPIRE servers authenticate to each other using their own SPIFFE Verifiable Identity Documents (SVIDs).

   apiVersion: operator.openshift.io/v1alpha1
   kind: SpireServer
   metadata:
     name: cluster
   spec:
     trustDomain: cluster1.example.com
     federation:
       bundleEndpoint:
         profile: https_spiffe
         refreshHint: 300
       managedRoute: "true"

   • The spec.trustDomain field sets a unique trust domain for each cluster.
   • The spec.federation.bundleEnpoint.profile field uses the https_spiffe profile for SPIFFE-based TLS authentication.
   • The spec.federation.bundleEndpoint.refreshHint field suggests intervals (in seconds) for remote servers to refresh the trust bundle. Range: 60-3600 seconds.
   • The spec.federation.managedRoute field enables automatic route creation by the Operator.

2. Apply the configuration changes by running the following command:

   $ oc apply -f spire-server.yaml

3. Check the status of the SPIRE Server by entering the following command. Wait for the Ready status to be returned.

   $ oc get spireserver cluster -w

4. Verify that the federation route has been created:

   $ oc get route -n zero-trust-workload-identity-manager | grep federation

   Example output

   NAME                      HOST/PORT                                    PATH   SERVICES        PORT    TERMINATION
   spire-server-federation   federation.apps.cluster1.example.com               spire-server     8443    passthrough

5. Fetch the trust bundle from each remote cluster’s federation endpoint:

   $  curl -k https://federation.apps.cluster2.example.com > cluster2-bundle.json

   Note

   For https_spiffe profile, you might need to use the -k flag if the certificate is not trusted by your system’s CA bundle:

   The response contains the trust bundle in JSON Web Key Set (JWKS) format:

   Example trust bundle

   {
     "keys": [
       {
         "use": "x509-svid",
         "kty": "RSA",
         "n": "...",
         "e": "AQAB",
         "x5c": ["..."]
       }
     ],
     "spiffe_sequence": 1,
     "refresh_hint": 300
   }

6. Create ClusterFederatedTrustDomain resources for each remote trust domain.

   1. On Cluster 1, create a resource to federate with Cluster 2:

      apiVersion: spire.spiffe.io/v1alpha1
      kind: ClusterFederatedTrustDomain
      metadata:
        name: federation-to-cluster2
      spec:
        trustDomain: <CLUSTER2_APPS_DOMAIN>
        bundleEndpointURL: https://federation.<CLUSTER2_APPS_DOMAIN>
        bundleEndpointProfile:
          type: https_spiffe
          endpointSPIFFEID: spiffe://<CLUSTER2_APPS_DOMAIN>/spire/server
        className: zero-trust-workload-identity-manager-spire
        trustDomainBundle: |
          {
            "keys": [
              {
                "use": "x509-svid",
                "kty": "RSA",
                "n": "...",
                "e": "AQAB",
                "x5c": ["..."]
              }
            ],
            "spiffe_sequence": 1
          }

      where

      <CLUSTER2_APPS_DOMAIN>

      Specifies the trust domain of the external cluster you are federating with.

      spec.bundleEndpointProfile.endpointSPIFFEID

      Specifies the SPIFFE ID of the remote SPIRE server. Required for https_spiffe profile to validate the remote server’s identity.

      spec.trustDomainBundle

      Specifies the complete trust bundle JSON that you fetched in the previous step.

      spec.className

      Specifies the name of a class to watch CRs for. Spire-controller-manager watches the resource only if spec.className is set to zero-trust-workload-identity-manager-spire.

7. Apply the ClusterFederatedTrustDomain resource by running the following command:

   $ oc apply -f clusterfederatedtrustdomain.yaml

8. Repeat steps 5-7 on each cluster for every remote cluster it should federate with. For bidirectional federation, each cluster needs a ClusterFederatedTrustDomain resource for every other cluster.

9. Update the SpireServer resource on each cluster to add the federatesWith configuration:

   apiVersion: operator.openshift.io/v1alpha1
   kind: SpireServer
   metadata:
     name: cluster
   spec:
     trustDomain: cluster1.example.com
     federation:
       bundleEndpoint:
         profile: https_spiffe
         refreshHint: 300
       federatesWith:
         - trustDomain: cluster2.example.com
           bundleEndpointUrl: https://federation.apps.cluster2.example.com
           bundleEndpointProfile: https_spiffe
           endpointSpiffeId: spiffe://cluster2.example.com/spire/server
         - trustDomain: cluster3.example.com
           bundleEndpointUrl: https://federation.apps.cluster3.example.com
           bundleEndpointProfile: https_spiffe
           endpointSpiffeId: spiffe://cluster3.example.com/spire/server
       managedRoute: "true"

   • The spec.federation.federatesWith field lists all remote trust domains this cluster should federate with.

10. Apply the updated configuration by running the following command:

    $ oc apply -f spireserver.yaml

Verification

1. Verify that the ClusterFederatedTrustDomain resources have been created by running the following command:

   $ oc get clusterfederatedtrustdomains

   Example output

   NAME                  TRUST DOMAIN           ENDPOINT URL                                      AGE
   cluster2-federation   cluster2.example.com   https://federation.apps.cluster2.example.com     5m
   cluster3-federation   cluster3.example.com   https://federation.apps.cluster3.example.com     5m

2. Check the status of a ClusterFederatedTrustDomain to ensure bundle synchronization is working by running the following command:

   $ oc describe clusterfederatedtrustdomain cluster2-federation

   Look for successful status conditions indicating that the trust bundle has been synchronized.

3. Verify that the federation endpoint is accessible by running the following command:

   $ curl https://federation.apps.cluster1.example.com

   You should receive a JSON response containing the trust bundle.

4. Check the SPIRE Server logs to confirm federation is active by running the following command:

   $ oc logs -n zero-trust-workload-identity-manager \
       statefulset/spire-server -c spire-server --tail=50

   Look for log messages indicating successful bundle synchronization with federated trust domains.

Procedure

1. Configure the SpireServer custom resource on each cluster to enable federation with ACME certificate management.

   Create or update your SpireServer resource with the federation configuration:

   apiVersion: operator.openshift.io/v1alpha1
   kind: SpireServer
   metadata:
     name: cluster
   spec:
     trustDomain: cluster1.example.com
     federation:
       bundleEndpoint:
         profile: https_web
         refreshHint: 300
         httpsWeb:
           acme:
             directoryUrl: https://acme-v02.api.letsencrypt.org/directory
             domainName: federation.apps.cluster1.example.com
             email: admin@example.com
             tosAccepted: "true"
       managedRoute: "true"

   • The spec.trustDomain field sets a unique trust domain for each cluster (for example, cluster1.example.com, cluster2.example.com).
   • The spec.federation.bundleEndpoint.profile field uses the https_web profile for ACME-based certificate management.
   • The spec.federation.bundleEndpoint.httpsWeb.acme.directoryUrl field contains the Let’s Encrypt production directory URL. For testing, use: https://acme-staging-v02.api.letsencrypt.org/directory.
   • The spec.federation.bundleEndpoint.httpsWeb.acme.domainName field is the domain name where your federation endpoint is accessible. This automatically sets to federation.<cluster-apps-domain> if managedRoute is set to "true".
   • The spec.federation.bundleEndpoint.httpsWeb.acme.email field is your email address for ACME account registration and certificate expiration notifications.
   • The spec.federation.bundleEndpoint.httpsWeb.acme.tosAccepted field accepts the Let’s Encrypt Terms of Service.
   • The spec.federation.managedRoute field enables an automatic route creation by the Operator for the federation bundle endpoint.

2. Apply the configuration to each cluster by running the following command:

   $ oc apply -f spireserver.yaml

3. Check the status of the SPIRE Server by entering the following command. Wait for the Ready status to be returned before proceeding to the next step.

   $ oc get spireserver cluster -w

   Example output

   NAME      STATUS   AGE
   cluster   Ready    5m

4. Verify that the federation route has been created by running the following command:

   $ oc get route -n zero-trust-workload-identity-manager | grep federation

   Example output

   NAME                      HOST/PORT                                          PATH   SERVICES        PORT   TERMINATION
   spire-server-federation   federation.apps.cluster1.example.com                     spire-server     8443    passthrough

5. On each cluster, fetch the trust bundle from the federation endpoint by running the following command:

   $ curl https://federation.apps.cluster1.example.com > cluster1-bundle.json

   The response contains the trust bundle in JSON Web Key Set (JWKS) format:

   Example trust bundle

   {
     "keys": [
       {
         "use": "x509-svid",
         "kty": "RSA",
         "n": "...",
         "e": "AQAB",
         "x5c": ["..."]
       }
     ],
     "spiffe_sequence": 1,
     "refresh_hint": 300
   }

6. Create ClusterFederatedTrustDomain resources to establish federation relationships.

   1. On Cluster 1, create resources to federate with Cluster 2 and Cluster 3:

      apiVersion: spire.spiffe.io/v1alpha1
      kind: ClusterFederatedTrustDomain
      metadata:
        name: cluster2-federation
      spec:
        trustDomain: cluster2.example.com
        bundleEndpointURL: https://federation.apps.cluster2.example.com
        bundleEndpointProfile:
          type: https_web
        className: zero-trust-workload-identity-manager-spire
        trustDomainBundle: |
          {
            "keys": [...],
            "spiffe_sequence": 1
          }
      ---
      apiVersion: spire.spiffe.io/v1alpha1
      kind: ClusterFederatedTrustDomain
      metadata:
        name: cluster3-federation
      spec:
        trustDomain: cluster3.example.com
        bundleEndpointURL: https://federation.apps.cluster3.example.com
        bundleEndpointProfile:
          type: https_web
        className: zero-trust-workload-identity-manager-spire
        trustDomainBundle: |
          {
            "keys": [...],
            "spiffe_sequence": 1
          }

      • The spec.trustDomainBundle field contains the complete trust bundle JSON that you fetched using curl in step 5.
      • The spec.className field contains the name of a class to watch CRs for. Spire-controller-manager watches the resource only if spec.className is set to zero-trust-workload-identity-manager-spire.

7. Apply the ClusterFederatedTrustDomain resources by running the following command:

   $ oc apply -f cluster-federated-trust-domains.yaml

8. Repeat steps 6 and 7 on each cluster to establish bidirectional federation. Each cluster needs ClusterFederatedTrustDomain resources for every other cluster it federates with.

9. Update the SpireServer resource on each cluster to add the federatesWith configuration:

   apiVersion: operator.openshift.io/v1alpha1
   kind: SpireServer
   metadata:
     name: cluster
   spec:
     # ... existing configuration ...
     federation:
       bundleEndpoint:
         # ... existing bundleEndpoint configuration ...
       federatesWith:
         - trustDomain: cluster2.example.com
           bundleEndpointUrl: https://federation.apps.cluster2.example.com
           bundleEndpointProfile: https_web
         - trustDomain: cluster3.example.com
           bundleEndpointUrl: https://federation.apps.cluster3.example.com
           bundleEndpointProfile: https_web
       managedRoute: "true"

   • The spec.federation.federatesWith field lists all remote trust domains this cluster should federate with.

10. Apply the updated configuration by running the following command:

    $ oc apply -f spireserver.yaml

Verification

1. Verify that the ClusterFederatedTrustDomain resources have been created by running the following command:

   $ oc get clusterfederatedtrustdomains

   Example output

   NAME                  TRUST DOMAIN          ENDPOINT URL                                   AGE
   cluster2-federation   cluster2.example.com  https://federation.apps.cluster2.example.com   5m
   cluster3-federation   cluster3.example.com  https://federation.apps.cluster3.example.com   5m

2. Check the status of a ClusterFederatedTrustDomain to ensure bundle synchronization is working by running the following command:

   $ oc describe clusterfederatedtrustdomain cluster2-federation

   Look for Successful status conditions indicating that the trust bundle has been synchronized.

3. Verify that the federation endpoint is accessible and serving the trust bundle by running the following command:

   $ curl https://federation.apps.cluster1.example.com

   You should receive a JSON response containing the trust bundle.

4. Check the SPIRE Server logs to confirm federation is active by running the following command:

   $ oc logs -n zero-trust-workload-identity-manager deployment/spire-server -c spire-server --tail=50

   Look for log messages indicating successful bundle synchronization with federated trust domains.

5. Verify that all SPIRE components are running by running the following command:

   $ oc get pods -n zero-trust-workload-identity-manager

   Example output

   NAME                    READY   STATUS    RESTARTS   AGE
   spire-agent-abcde       1/1     Running   0          10m
   spire-server-0          2/2     Running   0          10m

6. Optional: Test cross-cluster workload authentication by deploying workloads with SPIFFE identities on different clusters and verifying they can authenticate to each other using the federated trust.

Procedure

1. Install the cert-manager Operator on the cluster where you want to use externally managed certificates.

   Create a namespace and install the operator:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: cert-manager-operator
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-cert-manager-operator
     namespace: cert-manager-operator
   spec:
     upgradeStrategy: Default
   ---
   apiVersion: operators.coreos.com/stable-v1
   kind: Subscription
   metadata:
     name: openshift-cert-manager-operator
     namespace: cert-manager-operator
   spec:
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     name: openshift-cert-manager-operator
     channel: stable-v1

2. Apply the cert-manager installation by running the following command:

   $ oc apply -f cert-manager-install.yaml

3. Check the status of the cert-manager Operator by entering the following command:

   $ oc get pods -n cert-manager

   All cert-manager pods should be in Running status.

4. Create an Issuer for certificate provisioning.

   For Let’s Encrypt with HTTP-01 challenge:

   apiVersion: cert-manager.io/v1
   kind: Issuer
   metadata:
     name: letsencrypt-http01
     namespace: zero-trust-workload-identity-manager
   spec:
     acme:
       server: https://acme-v02.api.letsencrypt.org/directory
       privateKeySecretRef:
         name: letsencrypt-account-key
       solvers:
         - http01:
             ingress:
               ingressClassName: openshift-default

   Alternatively, for an internal CA:

   apiVersion: cert-manager.io/v1
   kind: Issuer
   metadata:
     name: internal-ca
     namespace: zero-trust-workload-identity-manager
   spec:
     ca:
       secretName: internal-ca-key-pair

5. Apply the Issuer by running the following command:

   $ oc apply -f issuer.yaml

6. Determine the federation endpoint domain name.

   The federation route follows a predictable naming pattern if managedRoute is set to true. Get your cluster’s application domain by running the following command:

   $ CLUSTER_DOMAIN=$(oc get ingresses.config/cluster -o jsonpath='{.spec.domain}')
   $ FEDERATION_DOMAIN="federation.${CLUSTER_DOMAIN}"
   $ echo "Federation domain will be: $FEDERATION_DOMAIN"

   Example output

   Federation domain will be: federation.apps.cluster1.example.com

   Note

   The federation route is created automatically if managedRoute is set to true when you apply the SpireServer configuration in a later step. The route name is spire-server-federation and the hostname is federation.<cluster-apps-domain>.

7. Create a Certificate resource to request a TLS certificate.

   Use the federation domain determined in the previous step:

   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: spire-server-federation-tls
     namespace: zero-trust-workload-identity-manager
   spec:
     secretName: spire-server-federation-tls
     duration: 2160h
     renewBefore: 360h
     commonName: federation.apps.cluster1.example.com
     dnsNames:
       - federation.apps.cluster1.example.com
     usages:
       - server auth
       - digital signature
       - key encipherment
     issuerRef:
       kind: Issuer
       name: letsencrypt-http01

   • The secretName field must match the externalSecretRef value in SpireServer.
   • The duration field shows how long a certificate is valid. Certificates are valid for 90 days.
   • The renewBefore field shows how many days a certificate must be renewed before it expires. Renew a certificate 15 days before expiration.
   • The commonName field must be replaced with your actual federation domain from the previous step.
   • The dnsNames field must match the commonName and the actual route hostname that was created.
   • The name field must reference the Issuer that was created earlier.

8. Apply the Certificate resource by running the following command:

   $ oc apply -f certificate.yaml

9. Monitor the certificate issuance by running the following command:

   $ oc get certificate spire-server-federation-tls \
       -n zero-trust-workload-identity-manager -w

   Example output when ready

   NAME                            READY   SECRET                          AGE
   spire-server-federation-tls     True    spire-server-federation-tls     2m

10. Create RBAC permissions for the OpenShift Ingress Router to access the certificate secret.

    Create a Role by running the following command:

    $ oc create role secret-reader \
        --verb=get,list,watch \
        --resource=secrets \
        --resource-name=spire-server-federation-tls \
        -n zero-trust-workload-identity-manager

    Create a RoleBinding by running the following command:

    $ oc create rolebinding secret-reader-binding \
        --role=secret-reader \
        --serviceaccount=openshift-ingress:router \
        -n zero-trust-workload-identity-manager

11. Configure the SpireServer custom resource to use manual certificate management.

    Now that the certificate is ready, configure the SpireServer to reference it:

    apiVersion: operator.openshift.io/v1alpha1
    kind: SpireServer
    metadata:
      name: cluster
    spec:
      trustDomain: cluster1.example.com
      federation:
        bundleEndpoint:
          profile: https_web
          refreshHint: 300
          httpsWeb:
            servingCert:
              fileSyncInterval: 86400
              externalSecretRef: spire-server-federation-tls
        managedRoute: "true"

    • The profile field must use https_web profile for certificate-based authentication.
    • The fileSyncInterval field checks for certificate updates every 24 hours (86400 seconds). Range: 3600-7776000 seconds.
    • The externalSecretRef field is the name of the secret containing the TLS certificate and private key. Must match the certificate secret created in the previous steps.

12. Apply the configuration by running the following command:

    $ oc apply -f spireserver.yaml

13. Wait for the SPIRE Server to be ready:

    $ oc get spireserver cluster -n zero-trust-workload-identity-manager -w

    Wait until the status shows Ready.

14. Verify that the federation route was created by running the following command:

    $ oc get route spire-server-federation -n zero-trust-workload-identity-manager

    Example output

    NAME                      HOST/PORT                                  PATH   SERVICES        PORT    TERMINATION
    spire-server-federation   federation.apps.cluster1.example.com              spire-server    8443    reencrypt

    Verify that the route hostname matches the domain name used in your certificate.

15. Verify that the federation endpoint is accessible by running the following command:

    $ curl https://$(oc get route spire-server-federation \
        -n zero-trust-workload-identity-manager \
        -o jsonpath='{.spec.host}')

    You should receive a JSON response containing the trust bundle.

16. Fetch the trust bundle from each federation endpoint that you want to federate with.

    For each remote cluster, fetch its trust bundle by running the following commands:

    $ curl https://federation.apps.cluster1.example.com > cluster1-bundle.json
    $ curl https://federation.apps.cluster2.example.com > cluster2-bundle.json

    The trust bundle is in JSON Web Key Set (JWKS) format:

    Example trust bundle

    {
      "keys": [
        {
          "use": "x509-svid",
          "kty": "RSA",
          "n": "xGOzB...",
          "e": "AQAB",
          "x5c": ["MIIC..."]
        }
      ],
      "spiffe_sequence": 1,
      "refresh_hint": 300
    }

17. Create ClusterFederatedTrustDomain resources for each remote trust domain you want to federate with:

    apiVersion: spire.spiffe.io/v1alpha1
    kind: ClusterFederatedTrustDomain
    metadata:
      name: cluster1-federation
    spec:
      trustDomain: cluster1.example.com
      bundleEndpointURL: https://federation.apps.cluster1.example.com
      bundleEndpointProfile:
        type: https_web
      className: zero-trust-workload-identity-manager-spire
      trustDomainBundle: |
        {
          "keys": [
            {
              "use": "x509-svid",
              "kty": "RSA",
              "n": "xGOzB...",
              "e": "AQAB",
              "x5c": ["MIIC..."]
            }
          ],
          "spiffe_sequence": 1
        }
    ---
    apiVersion: spire.spiffe.io/v1alpha1
    kind: ClusterFederatedTrustDomain
    metadata:
      name: cluster2-federation
    spec:
      trustDomain: cluster2.example.com
      bundleEndpointURL: https://federation.apps.cluster2.example.com
      bundleEndpointProfile:
        type: https_web
      className: zero-trust-workload-identity-manager-spire
      trustDomainBundle: |
        {
          "keys": [...],
          "spiffe_sequence": 1
        }

    • The trustDomainBundle field contains the complete trust bundle JSON that you fetched in the previous step.
    • The spec.className field contains the name of a class to watch CRs for. Spire-controller-manager watches the resource only if spec.className is set to zero-trust-workload-identity-manager-spire.

18. Apply the ClusterFederatedTrustDomain resources by running the following command:

    $ oc apply -f clusterfederatedtrustdomains.yaml

19. Update the SpireServer resource to add the federatesWith configuration:

    apiVersion: operator.openshift.io/v1alpha1
    kind: SpireServer
    metadata:
      name: cluster
    spec:
      trustDomain: cluster3.example.com
      federation:
        bundleEndpoint:
          profile: https_web
          refreshHint: 300
          httpsWeb:
            servingCert:
              fileSyncInterval: 86400
              externalSecretRef: spire-server-federation-tls
        federatesWith:
          - trustDomain: cluster1.example.com
            bundleEndpointUrl: https://federation.apps.cluster1.example.com
            bundleEndpointProfile: https_web
    
          - trustDomain: cluster2.example.com
            bundleEndpointUrl: https://federation.apps.cluster2.example.com
            bundleEndpointProfile: https_web
        managedRoute: "true"

    • The federatesWith field lists all remote trust domains this cluster should federate with.

20. Apply the updated configuration by running the following command:

    $ oc apply -f spireserver.yaml

21. Repeat steps 1-15 on each cluster that participates in the federation, ensuring that:

    • Each cluster has cert-manager installed and configured
    • Each cluster has its own certificate created and ready before applying the SpireServer configuration
    • Each cluster has the RBAC for the ingress router configured
    • Each cluster has ClusterFederatedTrustDomain resources for every other cluster it federates with
    • Each cluster’s SpireServer has the complete federatesWith list

Verification

1. Verify that the certificate has been issued successfully by running the following command:

   $ oc get certificate spire-server-federation-tls \
       -n zero-trust-workload-identity-manager

   Example output

   NAME                            READY   SECRET                          AGE
   spire-server-federation-tls     True    spire-server-federation-tls     5m

2. Check the certificate details and expiration by running the following command:

   $ oc get secret spire-server-federation-tls \
       -n zero-trust-workload-identity-manager \
       -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

   Example output

   notBefore=Dec 16 10:00:00 2025 GMT
   notAfter=Mar 16 10:00:00 2026 GMT

3. Verify that the RBAC permissions are configured correctly by running the following command:

   $ oc get role,rolebinding -n zero-trust-workload-identity-manager \
       | grep secret-reader

   Example output

   role.rbac.authorization.k8s.io/secret-reader
   rolebinding.rbac.authorization.k8s.io/secret-reader-binding

   Verify the RoleBinding references the correct ServiceAccount by running the following command:

   $ oc describe rolebinding secret-reader-binding \
       -n zero-trust-workload-identity-manager

   Example output

   Name:         secret-reader-binding
   Namespace:    zero-trust-workload-identity-manager
   Role:
     Kind:  Role
     Name:  secret-reader
   Subjects:
     Kind            Name    Namespace
     ----            ----    ---------
     ServiceAccount  router  openshift-ingress

4. Verify that the ClusterFederatedTrustDomain resources have been created by running the following command:

   $ oc get clusterfederatedtrustdomains

   Example output

   NAME                  TRUST DOMAIN           ENDPOINT URL                                      AGE
   cluster1-federation   cluster1.example.com   https://federation.apps.cluster1.example.com     5m
   cluster2-federation   cluster2.example.com   https://federation.apps.cluster2.example.com     5m

5. Check the status of a ClusterFederatedTrustDomain to ensure bundle synchronization is working by running the following command:

   $ oc describe clusterfederatedtrustdomain cluster1-federation

   Look for successful status conditions indicating that the trust bundle has been synchronized.

6. Verify that the federation endpoint is accessible and using the correct certificate by running the following command:

   $ curl -v https://$(oc get route spire-server-federation \
       -n zero-trust-workload-identity-manager \
       -o jsonpath='{.spec.host}')

   In the output, verify that the certificate presented is issued by your configured CA (Let’s Encrypt or internal CA).

7. Check the SPIRE Server logs to confirm that by running the following command:

   • Federation is active with remote trust domains
   • Trust bundles are being synchronized

   • The bundle endpoint is serving correctly

     $ oc logs -n zero-trust-workload-identity-manager \
         statefulset/spire-server -c spire-server --tail=100

     Look for log messages indicating successful federation bundle synchronization.

8. Verify that all SPIRE components are running by running the following command:

   $ oc get pods -n zero-trust-workload-identity-manager

   Example output

   NAME                                    READY   STATUS    RESTARTS   AGE
   spire-agent-abc123                      1/1     Running   0          10m
   spire-server-0                          2/2     Running   0          10m

9. Optional: Test cross-cluster workload authentication by deploying workloads with SPIFFE identities on different clusters and verifying they can authenticate to each other using the federated trust.

Procedure

1. Create the cluster-monitoring-config.yaml file to define and configure the ConfigMap:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       enableUserWorkload: true

2. Apply the ConfigMap by running the following command:

   $ oc apply -f cluster-monitoring-config.yaml

Procedure

1. Create the ServiceMonitor CR:

   1. Create the YAML file that defines the ServiceMonitor CR:

      Example servicemonitor-spire-server file

      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      metadata:
      labels:
        app.kubernetes.io/name: server
        app.kubernetes.io/instance: spire
      name: spire-server-metrics
      namespace: zero-trust-workload-identity-manager
      spec:
      endpoints:
      - port: metrics
        interval: 30s
        path: /metrics
      selector:
        matchLabels:
          app.kubernetes.io/name: server
          app.kubernetes.io/instance: spire
      namespaceSelector:
        matchNames:
        - zero-trust-workload-identity-manager

   2. Create the ServiceMonitor CR by running the following command:

      $ oc create -f servicemonitor-spire-server.yaml

      After the ServiceMonitor CR is created, the user workload Prometheus instance begins metrics collection from the SPIRE Server. The collected metrics are labeled with job="spire-server".

Verification

1. In the OpenShift Container Platform web console, navigate to Observe Targets.

2. In the Label filter field, enter the following label to filter the metrics targets:

   $ service=zero-trust-workload-identity-manager-metrics-service

3. Confirm that the Status column shows Up for the spire-server-metrics entry.

Procedure

1. Create the ServiceMonitor CR:

   1. Create the YAML file that defines the ServiceMonitor CR:

      Example servicemonitor-spire-agent.yaml file

      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      metadata:
        labels:
          app.kubernetes.io/name: agent
          app.kubernetes.io/instance: spire
        name: spire-agent-metrics
        namespace: zero-trust-workload-identity-manager
      spec:
        endpoints:
        - port: metrics
          interval: 30s
          path: /metrics
        selector:
          matchLabels:
            app.kubernetes.io/name: agent
            app.kubernetes.io/instance: spire
        namespaceSelector:
          matchNames:
          - zero-trust-workload-identity-manager

   2. Create the ServiceMonitor CR by running the following command:

      $ oc create -f servicemonitor-spire-agent.yaml

      After the ServiceMonitor CR is created, the user workload Prometheus instance begins metrics collection from the SPIRE Agent. The collected metrics are labeled with job="spire-agent".

Verification

1. In the OpenShift Container Platform web console, navigate to Observe Targets.

2. In the Label filter field, enter the following label to filter the metrics targets:

   $ service=spire-agent

3. Confirm that the Status column shows Up for the spire-agent-metrics entry.

Procedure

1. Configure the Operator to use HTTP or HTTPS protocols for the metrics server.

   1. Update the subscription object for Zero Trust Workload Identity Manager to configure the HTTP protocol by running the following command:

      $ oc -n zero-trust-workload-identity-manager patch subscription zero-trust-workload-identity-manager-subscription --type='merge' -p '{"spec":{"config":{"env":[{"name":"METRICS_BIND_ADDRESS","value":":8080"}, {"name": "METRICS_SECURE", "value": "false"}]}}}'

   2. Verify the Zero Trust Workload Identity Manager pod is redeployed and that the configured values for METRICS_BIND_ADDRESS and METRICS_SECURE is updated by running the following command:

      $ oc set env --list deployment/zero-trust-workload-identity-manager-controller-manager -n zero-trust-workload-identity-manager | grep -e METRICS_BIND_ADDRESS -e METRICS_SECURE -e container

      Example output

      deployments/zero-trust-workload-identity-manager-controller-manager, container manager
      METRICS_BIND_ADDRESS=:8080
      METRICS_SECURE=false

2. Create the Secret resource with kubernetes.io/service-account.name annotation to inject the token required for authenticating with the metrics server.

   1. Create the secret-zero-trust-workload-identity-manager.yaml YAML file:

      apiVersion: v1
      kind: Secret
      metadata:
       labels:
         name: zero-trust-workload-identity-manager
       name: zero-trust-workload-identity-manager-metrics-auth
       namespace: zero-trust-workload-identity-manager
       annotations:
         kubernetes.io/service-account.name: zero-trust-workload-identity-manager-controller-manager
      type: kubernetes.io/service-account-token

   2. Create the Secret resource by running the following command:

      $ oc apply -f secret-zero-trust-workload-identity-manager.yaml

3. Create the ClusterRoleBinding resource required for granting permissions to access the metrics.

   1. Create the clusterrolebinding-zero-trust-workload-identity-manager.yaml YAML file:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
       labels:
         name: zero-trust-workload-identity-manager
       name: zero-trust-workload-identity-manager-allow-metrics-access
      roleRef:
       apiGroup: rbac.authorization.k8s.io
       kind: ClusterRole
       name: zero-trust-workload-identity-manager-metrics-reader
      subjects:
      - kind: ServiceAccount
        name: zero-trust-workload-identity-manager-controller-manager
        namespace: zero-trust-workload-identity-manager

   2. Create the ClusterRoleBinding resource by running the following command:

      $ oc apply -f clusterrolebinding-zero-trust-workload-identity-manager.yaml

4. Create the following ServiceMonitor CR if the metrics server is configured to use http.

   1. Create the servicemonitor-zero-trust-workload-identity-manager-http.yaml YAML file:

      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      metadata:
        labels:
          name: zero-trust-workload-identity-manager
        name: zero-trust-workload-identity-manager-metrics-monitor
        namespace: zero-trust-workload-identity-manager
      spec:
        endpoints:
          - authorization:
              credentials:
                name: zero-trust-workload-identity-manager-metrics-auth
                key: token
              type: Bearer
            interval: 60s
            path: /metrics
            port: metrics-http
            scheme: http
            scrapeTimeout: 30s
        namespaceSelector:
          matchNames:
            - zero-trust-workload-identity-manager
        selector:
          matchLabels:
            name: zero-trust-workload-identity-manager

   2. Create the ServiceMonitor CR by running the following command:

      $ oc apply -f servicemonitor-zero-trust-workload-identity-manager-http.yaml

5. Create the following ServiceMonitor CR if the metrics server is configured to use https.

   1. Create the servicemonitor-zero-trust-workload-identity-manager-https.yaml YAML file:

      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      metadata:
        labels:
          name: zero-trust-workload-identity-manager
        name: zero-trust-workload-identity-manager-metrics-monitor
        namespace: zero-trust-workload-identity-manager
      spec:
        endpoints:
          - authorization:
              credentials:
                name: zero-trust-workload-identity-manager-metrics-auth
                key: token
              type: Bearer
            interval: 60s
            path: /metrics
            port: metrics-https
            scheme: https
            scrapeTimeout: 30s
            tlsConfig:
              ca:
                configMap:
                  name: openshift-service-ca.crt
                  key: service-ca.crt
              serverName: zero-trust-workload-identity-manager-metrics-service.zero-trust-workload-identity-manager.svc.cluster.local
        namespaceSelector:
          matchNames:
            - zero-trust-workload-identity-manager
        selector:
          matchLabels:
            name: zero-trust-workload-identity-manager

   2. Create the ServiceMonitor CR by running the following command:

      $ oc apply -f servicemonitor-zero-trust-workload-identity-manager-https.yaml

      After the ServiceMonitor CR is created, the user workload Prometheus instance begins metrics collection from the SPIRE Server. The collected metrics are labeled with job="zero-trust-workload-identity-manager-metrics-service".

Verification

1. In the OpenShift Container Platform web console, navigate to Observe Targets.

2. In the Label filter field, enter the following label to filter the metrics targets:

   $ service=zero-trust-workload-identity-manager-metrics-service

3. Confirm that the Status column shows Up for the zero-trust-workload-identity-manager entry.

Procedure

1. In the OpenShift Container Platform web console, navigate to Observe Metrics.

2. In the query field, enter the following PromQL expression to query SPIRE Server metrics:

   {job="spire-server"}

3. In the query field, enter the following PromQL expression to query SPIRE Agent metrics.

   {job="spire-agent"}

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Uninstall the Zero Trust Workload Identity Manager.

   1. Go to Ecosystem Installed Operators.
   2. Click the Options menu next to the Zero Trust Workload Identity Manager entry, and then click Uninstall Operator.
   3. In the confirmation dialog, click Uninstall.

Procedure

1. Uninstall the operands by running each of the following commands:

   1. Delete the SpireOIDCDiscoveryProvider cluster by running the following command:

      $ oc delete SpireOIDCDiscoveryProvider cluster

   2. Delete the SpiffeCSIDriver cluster by running the following command:

      $ oc delete SpiffeCSIDriver cluster -l=app.kubernetes.io/name=zero-trust-workload-identity-manager

   3. Delete the SpireAgent cluster by running the following command:

      $ oc delete SpireAgent cluster

   4. Delete the SpireServer cluster by running the following command:

      $ oc delete SpireServer cluster

   5. Delete the ZeroTrustWorkloadIdentityManager cluster by running the following command:

      $ oc delete ZeroTrustWorkloadIdentityManager cluster

   6. Delete the persistent volume claim (PVC) by running the following command:

      $ oc delete pvc -l=app.kubernetes.io/name=spire-server

   7. Delete the service by running the following command:

      $ oc delete service -l=app.kubernetes.io/name=zero-trust-workload-identity-manager -n zero-trust-workload-identity-manager

   8. Delete the namespace by running the following command:

      $ oc delete ns zero-trust-workload-identity-manager

   9. Delete the cluster role by running the following command:

      $ oc delete clusterrole -l=app.kubernetes.io/name=zero-trust-workload-identity-manager

   10. Delete the admission wehhook configuration by running the following command:

       $ oc delete validatingwebhookconfigurations -l=app.kubernetes.io/name=zero-trust-workload-identity-manager

2. Delete the custom resource definitions (CRDs) by running each of the following commands:

   1. Delete the SPIRE Server CRD by running the following command:

      $ oc delete crd spireservers.operator.openshift.io

   2. Delete the SPIRE Agent CRD by running the following command:

      $ oc delete crd spireagents.operator.openshift.io

   3. Delete the SPIFFEE CSI Drivers CRD by running the following command:

      $ oc delete crd spiffecsidrivers.operator.openshift.io

   4. Delete the SPIRE OIDC Discovery Provider CRD by running the following command:

      $ oc delete crd spireoidcdiscoveryproviders.operator.openshift.io

   5. Delete the SPIRE and SPIFFE cluster federated trust domains CRD by running the following command:

      $ oc delete crd clusterfederatedtrustdomains.spire.spiffe.io

   6. Delete the cluster SPIFFE IDs CRD by running the following command:

      $ oc delete crd clusterspiffeids.spire.spiffe.io

   7. Delete the SPIRE and SPIFFE cluster static entries CRD by running the following command:

      $ oc delete crd clusterstaticentries.spire.spiffe.io

   8. Delete the Zero Trust Workload Identity Manager CRD by running the following command:

      $ oc delete crd zerotrustworkloadidentitymanagers.operator.openshift.io