Chapter 10. Zero Trust Workload Identity Manager

1. The SPIRE, SPIRE Agent, SPIFFE CSI Driver, and the SPIRE OIDC Discovery Provider operands are deployed and managed by Zero Trust Workload Identity Manager via associated customer resource definitions (CRDs).
2. Watches are then registered for relevant Kubernetes resources and the necessary SPIRE CRDs are applied to the cluster.
3. The CR for the ZeroTrustWorkloadIdentityManager resource named cluster is deployed and managed by a controller.

4. To deploy the SPIRE Server, SPIRE Agent, SPIFFE CSI Driver, and SPIRE OIDC Discovery Provider, you need to create a custom resource of a each certain type and name it cluster. The custom resource types are as follows:

   • SPIRE Server - SpireServer
   • SPIRE Agent - SpireAgent
   • SPIFFE CSI Driver - SpiffeCSIDriver
   • SPIRE OIDC discovery provider - SpireOIDCDiscoveryProvider
5. When a node starts, the SPIRE Agent initializes, and connects to the SPIRE Server.
6. The SPIRE Agent begins the node attestation process. The agent collects information on the node’s identity such as label name and namespace. The agent securely provides the information it gathered through the attestation to the SPIRE Server.
7. The SPIRE Server then evaluates this information against its configured attestation policies and registration entries. If successful, the server generates an agent SVID and the Trust Bundle (CA Certificate) and securely sends this back to the SPIRE Agent.
8. A workload starts on the node and needs a secure identity. The workload connects to the agent’s Workload API and requests a SVID.
9. The SPIRE Agent receives the request and begins a workload attestation to gather information about the workload.
10. After the SPIRE Agent gathers the information, the information is sent to the SPIRE Server and the server checks its configured registration entries.
11. The SPIRE Agent receives the workload SVID and Trust Bundle and passes it on to the workload. The workload can now present their SVIDs to other SPIFFE-aware devices to communicate with them.

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Go to Ecosystem Software Catalog.
3. Search for Zero Trust Workload Identity Manager.

4. On the Install Operator page:

   1. Update the Update channel, if necessary. The channel defaults to stable-v1, which installs the latest stable-v1 release of the Zero Trust Workload Identity Manager.

   2. Choose the Installed Namespace for the Operator. The default Operator namespace is zero-trust-workload-identity-manager.

      If the zero-trust-workload-identity-manager namespace does not exist, it is created for you.

      Note

      The Operator and operands are deployed in the same namespace.

   3. Select an Update Approval strategy

      • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
      • The Manual strategy requires a user with appropriate credentials to approve the Operator update.
5. Click Install.

Verification

1. Navigate to Ecosystem Installed Operators.

   1. Verify that Zero Trust Workload Identity Manager is listed with a Status of Succeeded in the zero-trust-workload-identity-manager namespace.

   2. Verify that Zero Trust Workload Identity Manager controller manager deployment is ready and available by running the following command:

      $ oc get deployment -l name=zero-trust-workload-identity-manager -n zero-trust-workload-identity-manager

      Example output

      NAME                                                      READY   UP-TO-DATE   AVAILABLE   AGE
      zero-trust-workload-identity-manager-controller-manager   1/1     1            1           3h36m

2. To check the Operator logs, run the following command:

   $ oc logs -f deployment/zero-trust-workload-identity-manager-controller-manager -n zero-trust-workload-identity-manager

Procedure

1. Create a new project named zero-trust-workload-identity-manager by running the following command:

   $ oc new-project zero-trust-workload-identity-manager

2. Create an OperatorGroup object:

   1. Create a YAML file, for example, operatorGroup.yaml, with the following content:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: openshift-zero-trust-workload-identity-manager
        namespace: zero-trust-workload-identity-manager
      spec:
        upgradeStrategy: Default

   2. Create the OperatorGroup object by running the following command:

      $ oc create -f operatorGroup.yaml

3. Create a Subscription object:

   1. Create a YAML file, for example, subscription.yaml, that defines the Subscription object:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: openshift-zero-trust-workload-identity-manager
        namespace: zero-trust-workload-identity-manager
      spec:
        channel: stable-v1
        name: openshift-zero-trust-workload-identity-manager
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        installPlanApproval: Automatic

   2. Create the Subscription object by running the following command:

      $ oc create -f subscription.yaml

Procedure

1. Create the SpireServer CR:

   1. Create a YAML file that defines the SpireServer CR, for example, SpireServer.yaml:

      The following is an example of a SpireServer.yaml file.

      apiVersion: operator.openshift.io/v1alpha1
      kind: SpireServer
      metadata:
       name: cluster
      spec:
        logLevel: "info"
        logFormat: "text"
        jwtIssuer: "https://oidc-discovery.apps.cluster.example.com"
        caValidity: "24h"
        defaultX509Validity: "1h"
        defaultJWTValidity: "5m"
        jwtKeyType: "rsa-2048"
        caSubject:
          country: "US"
          organization: "Example Corporation"
          commonName: "SPIRE Server CA"
        persistence:
          size: "5Gi"
          accessMode: "ReadWriteOnce"
          storageClass: "gp3-csi"
        datastore:
          databaseType: "sqlite3"
          connectionString: "/run/spire/data/datastore.sqlite3"
          tlsSecretName: ""
          maxOpenConns: 100
          maxIdleConns: 10
          connMaxLifetime: 0
          disableMigration: "false"

      where:

      metadata.name

      Specifies that the value must be cluster.

      spec.logLevel

      Specifies the logging level for the SPIRE Server. The valid options are debug, info, warn, and error.

      spec.logFormat

      Specifies the logging format for the SPIRE Server. The valid options are text and json.

      spec.jwtIssuer

      Specifies the JWT issuer URL. Must be a valid HTTPS or HTTP URL with a maximum length of 512 characters.

      spec.caValidity

      Specifies the validity period (Time to Live (TTL)) for the SPIRE Server’s CA certificate. This determines how long the server’s root or intermediate certificate is valid. The format is a duration string (for example, 24h, 168h).

      spec.defaultX509Validity

      Specifies the default validity period (TTL) for X.509 SVIDs issued to workloads. This value is used if a specific TTL is not configured for a registration entry.

      spec.defaultJWTValidity

      Specifies thedefault validity period (TTL) for JWT SVIDs issued to workloads. This value is used if a specific TTL is not configured for a registration entry.

      spec.wtKeyType

      Specifies the key type used for JWT signing. The valid options are rsa-2048, rsa-4096, ec-p256, and ec-p384. This field is optional.

      spec.caSubject.country

      Specifies the country for the SPIRE Server certificate authority (CA). Must be an ISO 3166-1 alpha-2 country code (2 characters).

      spec.caSubject.organization

      Specifies the organization for the SPIRE Server CA. Maximum length is 64 characters.

      spec.caSubject.commonName

      Specifies the common name for the SPIRE Server CA. Maximum length is 255 characters.

      spec.persistence.size

      Specifies the size of the persistent volume (for example, 1Gi, 5Gi). Once set, this field is immutable.

      spec.persistence.accessMode

      Specifies the access mode for the persistent volume. The valid options are ReadWriteOnce, ReadWriteOncePod, and ReadWriteMany. Once set, this field is immutable.

      spec.persistence.storageClass

      Specifies the storage class to be used for the PVC. Once set, this field is immutable.

      spec.datastore.databaseType

      Specifies the type of database to use for the datastore. The valid options are sql, sqlite3, postgres, mysql, aws_postgresql, and aws_mysql.

      spec.datastore.connectionString

      Specifies the connection string for the database. For PostgreSQL with SSL, include sslmode and certificate paths (for example, dbname=spire user=spire host=postgres.example.com sslmode=verify-full).

      spec.datastore.tlsSecretName

      Specifies the name of a Kubernetes Secret containing TLS certificates for database connections. The Secret will be mounted at /run/spire/db/certs. This field is optional.

      spec.datastore.maxOpenConns

      Specifies the maximum number of open database connections. Must be between 1 and 10000.

      spec.datastore.maxIdleConns

      Specifies the maximum number of idle database connections in the pool. Must be between 0 and 10000.

      spec.datastore.connMaxLifetime

      Specifies the maximum lifetime of a database connection in seconds. A value of 0 means connections are not closed due to age.

      spec.datastore.disableMigration

      Specifies whether to disable automatic database migration. The valid options are true and false.

   2. Apply the configuration by running the following command:

      $ oc apply -f SpireServer.yaml

Procedure

1. Create the SpireAgent CR:

   1. Create a YAML file that defines the SpireAgent CR, for example, SpireAgent.yaml:

      The following is an example of a SpireAgent.yaml file.

      apiVersion: operator.openshift.io/v1alpha1
      kind: SpireAgent
      metadata:
       name: cluster
      spec:
        socketPath: "/run/spire/agent-sockets"
        logLevel: "info"
        logFormat: "text"
        nodeAttestor:
          k8sPSATEnabled: "true"
        workloadAttestors:
          k8sEnabled: "true"
          workloadAttestorsVerification:
            type: "auto"
            hostCertBasePath: "/etc/kubernetes"
            hostCertFileName: "kubelet-ca.crt"
          disableContainerSelectors: "false"
          useNewContainerLocator: "true"

      where:

      metadata.name

      Specifies that the value must be cluster.

      spec.socketPath

      Specifies the directory on the host where the SPIRE agent socket is created. This directory is shared with the SPIFFE CSI driver via the hostPath volume. Must match the SpiffeCSIDriver.spec.agentSocketPath for workloads to access the socket. Must be an absolute path with a maximum length of 256 characters.

      spec.logLevel

      Specifies the logging level for the SPIRE Server. The valid options are debug, info, warn, and error.

      spec.logFormat

      Specifies the logging format for the SPIRE Server. The valid options are text and json.

      spec.nodeAttestor.k8sPSATEnabled

      Specifies whether Kubernetes Projected Service Account Token (PSAT) node attestation is enabled. When enabled, the SPIRE agent uses K8s PSATs to prove its identity to the SPIRE server during node attestation. The valid options are true and false.

      spec.workloadAttestors.k8sEnabled

      Specifies whether the Kubernetes workload attestor is enabled. When enabled, the SPIRE agent can verify workload identities using Kubernetes pod information and service account tokens. The valid options are true and false.

      spec.workloadAttestors.workloadAttestorsVerification.type

      Specifies the kubelet certificate verification mode. The valid options are auto, hostCert, and skip.

      spec.workloadAttestors.workloadAttestorsVerification.hostCertBasePath

      Specifies the directory containing the kubelet CA certificate. Required when type is hostCert. Optional when type is auto (defaults to /etc/kubernetes if not specified).

      spec.workloadAttestors.workloadAttestorsVerification.hostCertFileName

      Specifies the file name for the kubelet’s CA certificate. When combined with hostCertBasePath, forms the full path. Required when type is hostCert. Optional when type is auto. Defaults to kubelet-ca.crt if not specified.

      spec.workloadAttestors.disableContainerSelectors

      Specifies whether to disable container selectors in the Kubernetes workload attestor. Set to true if using holdApplicationUntilProxyStarts in Istio. The valid options are true and false.

      spec.workloadAttestors.useNewContainerLocator

      Specifies enabling the new container locator algorithm that has support for cgroups v2. The valid options are true and false.

   2. Apply the configuration by running the following command:

      $ oc apply -f SpireAgent.yaml

Procedure

1. Create the SpiffeCSIDriver CR:

   1. Create a YAML file that defines the SpiffeCSIDriver CR object, for example, SpiffeCSIDriver.yaml:

      Example SpiffeCSIDriver.yaml

      apiVersion: operator.openshift.io/v1alpha1
      kind: SpiffeCSIDriver
      metadata:
       name: cluster
      spec:
        agentSocketPath: "/run/spire/agent-sockets"
        pluginName: "csi.spiffe.io"

      where:

      metadata.name

      Specifies that the name must be cluster.

      spec.agentSocketPath

      Specifies the path to the directory containing the SPIRE agent’s Workload API socket. This directory is bind-mounted into workload containers by the CSI driver. The directory is shared between the SPIRE agent and CSI driver via a hostPath volume. Must be an absolute path with a maximum length of 256 characters. This value must match SpireAgent.spec.socketPath for workloads to access the socket.

      spec.pluginName

      Specifies the name of the CSI plugin. This sets the CSI driver name that is deployed to the cluster and used in VolumeMount configurations. Must match the driver name referenced in the workload pods. Must be a valid domain name format (for example, csi.spiffe.io) with a maximum length of 127 characters.

   2. Apply the configuration by running the following command:

      $ oc apply -f SpiffeCSIDriver.yaml

Procedure

1. Create the SpireOIDCDiscoveryProvider CR:

   1. Create a YAML file that defines the SpireOIDCDiscoveryProvider CR, for example, SpireOIDCDiscoveryProvider.yaml:

      Example SpireOIDCDiscoveryProvider YAML

      aapiVersion: operator.openshift.io/v1alpha1
      kind: SpireOIDCDiscoveryProvider
      metadata:
       name: cluster
      spec:
        logLevel: "info"
        logFormat: "text"
        csiDriverName: "csi.spiffe.io"
        jwtIssuer: "https://oidc-discovery.apps.cluster.example.com"
        replicaCount: 1
        managedRoute: "true"
        externalSecretRef: ""

      where:

      metadata.name

      Specifies that the value must be cluster.

      spec.logLevel

      Specifies the logging level for the SPIRE Server. The valid options are debug, info, warn, and error.

      spec.logFormat

      Specifies the logging format for the SPIRE Server. The valid options are text and json.

      spec.csiDriverName

      Specifies the name of the CSI driver to use for mounting the Workload API socket. This must match the SpiffeCSIDriver.spec.pluginName value for the OIDC provider to access SPIFFE identities. Must be a valid DNS subdomain format (for example, csi.spiffe.io) with a maximum length of 127 characters.

      spec.jwtIssuer

      Specifies the JWT issuer URL. Must be a valid HTTPS or HTTP URL with a maximum length of 512 characters. This value must match the SpireServer.spec.jwtIssuer value.

      spec.replicaCount

      Specifies the number of replicas for the OIDC Discovery Provider deployment. Must be between 1 and 5.

      spec.managedRoute

      Specifies whether the Operator automatically creates an OpenShift route for the OIDC Discovery Provider endpoints. Set to true to have the Operator automatically create and maintain an OpenShift route for OIDC discovery endpoints (*.apps.). Set to false for administrators to manually configure routes or ingress.

      spec.externalSecretRef

      Specifies a reference to an externally managed secret that contains the TLS certificate for the OIDC Discovery Provider route host. Must be a valid Kubernetes secret reference name with a maximum length of 253 characters. This field is optional.

   2. Apply the configuration by running the following command:

      $ oc apply -f SpireOIDCDiscoveryProvider.yaml

Verification

1. Verify that the deployment of OIDC Discovery Provider is ready and available by running the following command:

   $ oc get deployment -l app.kubernetes.io/name=spiffe-oidc-discovery-provider -n zero-trust-workload-identity-manager

   Example output

   NAME                                    READY  UP-TO-DATE  AVAILABLE  AGE
   spire-spiffe-oidc-discovery-provider    1/1    1           1          2m58s

2. Verify that the status of OIDC Discovery Provider pods is Running by running the following command:

   $ oc get po -l app.kubernetes.io/name=spiffe-oidc-discovery-provider -n zero-trust-workload-identity-manager

   Example output

   NAME                                                    READY   STATUS    RESTARTS   AGE
   spire-spiffe-oidc-discovery-provider-64586d599f-lcc94   2/2     Running   0          7m15s

Procedure

1. Confirm the SPIFFE CSI driver is ready by running the following command:

   $ oc get spiffecsidriver cluster -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'

   The expected output includes DaemonSetAvailable=True and Ready=True.

2. Confirm that the CSI DaemonSet is available by running the following command:

   $ oc get ds spire-spiffe-csi-driver -n zero-trust-workload-identity-manager

3. Confirm the CSI pods are running and are using the privileged SCC by running the following commands:

   $ oc get pods -n zero-trust-workload-identity-manager -l app.kubernetes.io/name=spiffe-csi-driver

   $ oc get pod -n zero-trust-workload-identity-manager -l app.kubernetes.io/name=spiffe-csi-driver \
       -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.openshift\.io/scc}{"\n"}{end}'

   Every pod should show privileged.

4. After all checks have passed, delete the legacy custom SCC by running the following commands:

   $ oc get scc spire-spiffe-csi-driver

   $ oc delete scc spire-spiffe-csi-driver

   If the SCC is already absent, no action is needed.

Procedure

1. Create a config map in the zero-trust-workload-identity-manager namespace by running the following command:

   $ oc create configmap trusted-ca -n zero-trust-workload-identity-manager

2. Inject the CA bundle that is trusted by OpenShift Container Platform into the config map by running the following command:

   $ oc label cm trusted-ca config.openshift.io/inject-trusted-cabundle=true -n zero-trust-workload-identity-manager

3. Update the subscription for the Zero Trust Workload Identity Manager to use the config map by running the following command:

   $ oc -n zero-trust-workload-identity-manager patch subscription openshift-zero-trust-workload-identity-manager --type='merge' -p '{"spec":{"config":{"env":[{"name":"TRUSTED_CA_BUNDLE_CONFIGMAP","value":"trusted-ca"}]}}}'

Verification

1. Verify that the operands have finished rolling out by running the following command:

   $ oc rollout status deployment/zero-trust-workload-identity-manager-controller-manager -n zero-trust-workload-identity-manager && \
   $ oc rollout status statefulset/spireserver -n zero-trust-workload-identity-manager && \
   $ oc rollout status daemonset/spire-agent -n zero-trust-workload-identity-manager && \
   $ oc rollout status deployment/spire-spiffe-oidc-discovery-provider -n zero-trust-workload-identity-manager

   Example output

   deployment "zero-trust-workload-identity-manager-controller-manager" successfully rolled out
   statefulset "spire-server" successfully rolled out
   daemonset "spire-agent" successfully rolled out
   deployment "spire-spiffe-oidc-discovery-provider" successfully rolled out

2. Verify that the CA bundle was mounted as a volume by running the following command:

   $ oc get deployment zero-trust-workload-identity-manager -n zero-trust-workload-identity-manager -o=jsonpath={.spec.template.spec.'containers[0].volumeMounts'}

   $ oc get statefulset spire-server -n zero-trust-workload-identity-manager -o jsonpath='{.spec.template.spec.containers[*].volumeMounts[?(@.name=="trusted-ca-bundle")]}'

   $ oc get daemonset spire-agent -n zero-trust-workload-identity-manager -o jsonpath='{.spec.template.spec.containers[*].volumeMounts[?(@.name=="trusted-ca-bundle")]}'

   $ oc get daemonset spire-spiffe-csi-driver -n zero-trust-workload-identity-manager -o jsonpath='{.spec.template.spec.containers[*].volumeMounts[?(@.name=="trusted-ca-bundle")]}'

   Example output

   [{{"mountPath":"/etc/pki/ca-trust/extracted/pem","name":"trusted-ca-bundle","readOnly":true}]

3. Verify that the source of the CA bundle is the trusted-ca config map by running the following command:

   $ oc get deployment zero-trust-workload-identity-manager -n zero-trust-workload-identity-manager -o=jsonpath={.spec.template.spec.volumes}

   $ oc get statefulset spire-server -n zero-trust-workload-identity-manager -o=jsonpath='{.spec.template.spec.volumes}' | jq '.[] | select(.name=="trusted-ca-bundle")'

   $ oc get daemonset spire-agent -n zero-trust-workload-identity-manager -o=jsonpath='{.spec.template.spec.volumes}' | jq '.[] | select(.name=="trusted-ca-bundle")'

   $ oc get deployment spire-spiffe-oidc-discovery-provider -n zero-trust-workload-identity-manager -o=jsonpath='{.spec.template.spec.volumes}' | jq '.[] | select(.name=="trusted-ca-bundle")'

   Example output

   {
     "configMap": {
       "defaultMode": 420,
       "items": [
         {
           "key": "ca-bundle.crt",
           "path": "tls-ca-bundle.pem"
         }
       ],
       "name": "trusted-ca"
     },
     "name": "trusted-ca-bundle"
   }

Procedure

1. Configure the SpireServer custom resource on each cluster to enable federation with the https_spiffe profile. The https_spiffe profile uses SPIFFE-based TLS authentication, where SPIRE servers authenticate to each other using their own SPIFFE Verifiable Identity Documents (SVIDs).

   apiVersion: operator.openshift.io/v1alpha1
   kind: SpireServer
   metadata:
     name: cluster
   spec:
     trustDomain: cluster1.example.com
     federation:
       bundleEndpoint:
         profile: https_spiffe
         refreshHint: 300
       managedRoute: "true"

   • The spec.trustDomain field sets a unique trust domain for each cluster.
   • The spec.federation.bundleEnpoint.profile field uses the https_spiffe profile for SPIFFE-based TLS authentication.
   • The spec.federation.bundleEndpoint.refreshHint field suggests intervals (in seconds) for remote servers to refresh the trust bundle. Range: 60-3600 seconds.
   • The spec.federation.managedRoute field enables automatic route creation by the Operator.

2. Apply the configuration changes by running the following command:

   $ oc apply -f spire-server.yaml

3. Check the status of the SPIRE Server by entering the following command. Wait for the Ready status to be returned.

   $ oc get spireserver cluster -w

4. Verify that the federation route has been created:

   $ oc get route -n zero-trust-workload-identity-manager | grep federation

   Example output

   NAME                      HOST/PORT                                    PATH   SERVICES        PORT    TERMINATION
   spire-server-federation   federation.apps.cluster1.example.com               spire-server     8443    passthrough

5. Fetch the trust bundle from each remote cluster’s federation endpoint:

   $  curl -k https://federation.apps.cluster2.example.com > cluster2-bundle.json

   Note

   For https_spiffe profile, you might need to use the -k flag if the certificate is not trusted by your system’s CA bundle:

   The response contains the trust bundle in JSON Web Key Set (JWKS) format:

   Example trust bundle

   {
     "keys": [
       {
         "use": "x509-svid",
         "kty": "RSA",
         "n": "...",
         "e": "AQAB",
         "x5c": ["..."]
       }
     ],
     "spiffe_sequence": 1,
     "refresh_hint": 300
   }

6. Create ClusterFederatedTrustDomain resources for each remote trust domain.

   1. On Cluster 1, create a resource to federate with Cluster 2:

      apiVersion: spire.spiffe.io/v1alpha1
      kind: ClusterFederatedTrustDomain
      metadata:
        name: federation-to-cluster2
      spec:
        trustDomain: <CLUSTER2_APPS_DOMAIN>
        bundleEndpointURL: https://federation.<CLUSTER2_APPS_DOMAIN>
        bundleEndpointProfile:
          type: https_spiffe
          endpointSPIFFEID: spiffe://<CLUSTER2_APPS_DOMAIN>/spire/server
        className: zero-trust-workload-identity-manager-spire
        trustDomainBundle: |
          {
            "keys": [
              {
                "use": "x509-svid",
                "kty": "RSA",
                "n": "...",
                "e": "AQAB",
                "x5c": ["..."]
              }
            ],
            "spiffe_sequence": 1
          }

      where

      <CLUSTER2_APPS_DOMAIN>

      Specifies the trust domain of the external cluster you are federating with.

      spec.bundleEndpointProfile.endpointSPIFFEID

      Specifies the SPIFFE ID of the remote SPIRE server. Required for https_spiffe profile to validate the remote server’s identity.

      spec.trustDomainBundle

      Specifies the complete trust bundle JSON that you fetched in the previous step.

      spec.className

      Specifies the name of a class to watch CRs for. Spire-controller-manager watches the resource only if spec.className is set to zero-trust-workload-identity-manager-spire.

7. Apply the ClusterFederatedTrustDomain resource by running the following command:

   $ oc apply -f clusterfederatedtrustdomain.yaml

8. Repeat steps 5-7 on each cluster for every remote cluster it should federate with. For bidirectional federation, each cluster needs a ClusterFederatedTrustDomain resource for every other cluster.

9. Update the SpireServer resource on each cluster to add the federatesWith configuration:

   apiVersion: operator.openshift.io/v1alpha1
   kind: SpireServer
   metadata:
     name: cluster
   spec:
     trustDomain: cluster1.example.com
     federation:
       bundleEndpoint:
         profile: https_spiffe
         refreshHint: 300
       federatesWith:
         - trustDomain: cluster2.example.com
           bundleEndpointUrl: https://federation.apps.cluster2.example.com
           bundleEndpointProfile: https_spiffe
           endpointSpiffeId: spiffe://cluster2.example.com/spire/server
         - trustDomain: cluster3.example.com
           bundleEndpointUrl: https://federation.apps.cluster3.example.com
           bundleEndpointProfile: https_spiffe
           endpointSpiffeId: spiffe://cluster3.example.com/spire/server
       managedRoute: "true"

   • The spec.federation.federatesWith field lists all remote trust domains this cluster should federate with.

10. Apply the updated configuration by running the following command:

    $ oc apply -f spireserver.yaml

Verification

1. Verify that the ClusterFederatedTrustDomain resources have been created by running the following command:

   $ oc get clusterfederatedtrustdomains

   Example output

   NAME                  TRUST DOMAIN           ENDPOINT URL                                      AGE
   cluster2-federation   cluster2.example.com   https://federation.apps.cluster2.example.com     5m
   cluster3-federation   cluster3.example.com   https://federation.apps.cluster3.example.com     5m

2. Check the status of a ClusterFederatedTrustDomain to ensure bundle synchronization is working by running the following command:

   $ oc describe clusterfederatedtrustdomain cluster2-federation

   Look for successful status conditions indicating that the trust bundle has been synchronized.

3. Verify that the federation endpoint is accessible by running the following command:

   $ curl https://federation.apps.cluster1.example.com

   You should receive a JSON response containing the trust bundle.

4. Check the SPIRE Server logs to confirm federation is active by running the following command:

   $ oc logs -n zero-trust-workload-identity-manager \
       statefulset/spire-server -c spire-server --tail=50

   Look for log messages indicating successful bundle synchronization with federated trust domains.

Procedure

1. Configure the SpireServer custom resource on each cluster to enable federation with ACME certificate management.

   Create or update your SpireServer resource with the federation configuration:

   apiVersion: operator.openshift.io/v1alpha1
   kind: SpireServer
   metadata:
     name: cluster
   spec:
     trustDomain: cluster1.example.com
     federation:
       bundleEndpoint:
         profile: https_web
         refreshHint: 300
         httpsWeb:
           acme:
             directoryUrl: https://acme-v02.api.letsencrypt.org/directory
             domainName: federation.apps.cluster1.example.com
             email: admin@example.com
             tosAccepted: "true"
       managedRoute: "true"

   • The spec.trustDomain field sets a unique trust domain for each cluster (for example, cluster1.example.com, cluster2.example.com).
   • The spec.federation.bundleEndpoint.profile field uses the https_web profile for ACME-based certificate management.
   • The spec.federation.bundleEndpoint.httpsWeb.acme.directoryUrl field contains the Let’s Encrypt production directory URL. For testing, use: https://acme-staging-v02.api.letsencrypt.org/directory.
   • The spec.federation.bundleEndpoint.httpsWeb.acme.domainName field is the domain name where your federation endpoint is accessible. This automatically sets to federation.<cluster-apps-domain> if managedRoute is set to "true".
   • The spec.federation.bundleEndpoint.httpsWeb.acme.email field is your email address for ACME account registration and certificate expiration notifications.
   • The spec.federation.bundleEndpoint.httpsWeb.acme.tosAccepted field accepts the Let’s Encrypt Terms of Service.
   • The spec.federation.managedRoute field enables an automatic route creation by the operator for the federation bundle endpoint.

2. Apply the configuration to each cluster by running the following command:

   $ oc apply -f spireserver.yaml

3. Check the status of the SPIRE Server by entering the following command. Wait for the Ready status to be returned before proceeding to the next step.

   $ oc get spireserver cluster -w

   Example output

   NAME      STATUS   AGE
   cluster   Ready    5m

4. Verify that the federation route has been created by running the following command:

   $ oc get route -n zero-trust-workload-identity-manager | grep federation

   Example output

   NAME                      HOST/PORT                                          PATH   SERVICES        PORT   TERMINATION
   spire-server-federation   federation.apps.cluster1.example.com                     spire-server     8443    passthrough

5. On each cluster, fetch the trust bundle from the federation endpoint by running the following command:

   $ curl https://federation.apps.cluster1.example.com > cluster1-bundle.json

   The response contains the trust bundle in JSON Web Key Set (JWKS) format:

   Example trust bundle

   {
     "keys": [
       {
         "use": "x509-svid",
         "kty": "RSA",
         "n": "...",
         "e": "AQAB",
         "x5c": ["..."]
       }
     ],
     "spiffe_sequence": 1,
     "refresh_hint": 300
   }

6. Create ClusterFederatedTrustDomain resources to establish federation relationships.

   1. On Cluster 1, create resources to federate with Cluster 2 and Cluster 3:

      apiVersion: spire.spiffe.io/v1alpha1
      kind: ClusterFederatedTrustDomain
      metadata:
        name: cluster2-federation
      spec:
        trustDomain: cluster2.example.com
        bundleEndpointURL: https://federation.apps.cluster2.example.com
        bundleEndpointProfile:
          type: https_web
        className: zero-trust-workload-identity-manager-spire
        trustDomainBundle: |
          {
            "keys": [...],
            "spiffe_sequence": 1
          }
      ---
      apiVersion: spire.spiffe.io/v1alpha1
      kind: ClusterFederatedTrustDomain
      metadata:
        name: cluster3-federation
      spec:
        trustDomain: cluster3.example.com
        bundleEndpointURL: https://federation.apps.cluster3.example.com
        bundleEndpointProfile:
          type: https_web
         className: zero-trust-workload-identity-manager-spire
        trustDomainBundle: |
          {
            "keys": [...],
            "spiffe_sequence": 1
          }

      • The spec.trustDomainBundle field contains the complete trust bundle JSON that you fetched using curl in step 5.
      • The spec.className field contains the name of a class to watch CRs for. Spire-controller-manager watches the resource only if spec.className is set to zero-trust-workload-identity-manager-spire.

7. Apply the ClusterFederatedTrustDomain resources by running the following command:

   $ oc apply -f cluster-federated-trust-domains.yaml

8. Repeat steps 6 and 7 on each cluster to establish bidirectional federation. Each cluster needs ClusterFederatedTrustDomain resources for every other cluster it federates with.

9. Update the SpireServer resource on each cluster to add the federatesWith configuration:

   apiVersion: operator.openshift.io/v1alpha1
   kind: SpireServer
   metadata:
     name: cluster
   spec:
     # ... existing configuration ...
     federation:
       bundleEndpoint:
         # ... existing bundleEndpoint configuration ...
       federatesWith:
         - trustDomain: cluster2.example.com
           bundleEndpointUrl: https://federation.apps.cluster2.example.com
           bundleEndpointProfile: https_web
         - trustDomain: cluster3.example.com
           bundleEndpointUrl: https://federation.apps.cluster3.example.com
           bundleEndpointProfile: https_web
       managedRoute: "true"

   • The spec.federation.federatesWith field lists all remote trust domains this cluster should federate with.

10. Apply the updated configuration by running the following command:

    $ oc apply -f spireserver.yaml

Verification

1. Verify that the ClusterFederatedTrustDomain resources have been created by running the following command:

   $ oc get clusterfederatedtrustdomains

   Example output

   NAME                  TRUST DOMAIN          ENDPOINT URL                                   AGE
   cluster2-federation   cluster2.example.com  https://federation.apps.cluster2.example.com   5m
   cluster3-federation   cluster3.example.com  https://federation.apps.cluster3.example.com   5m

2. Check the status of a ClusterFederatedTrustDomain to ensure bundle synchronization is working by running the following command:

   $ oc describe clusterfederatedtrustdomain cluster2-federation

   Look for Successful status conditions indicating that the trust bundle has been synchronized.

3. Verify that the federation endpoint is accessible and serving the trust bundle by running the following command:

   $ curl https://federation.apps.cluster1.example.com

   You should receive a JSON response containing the trust bundle.

4. Check the SPIRE Server logs to confirm federation is active by running the following command:

   $ oc logs -n zero-trust-workload-identity-manager deployment/spire-server -c spire-server --tail=50

   Look for log messages indicating successful bundle synchronization with federated trust domains.

5. Verify that all SPIRE components are running by running the following command:

   $ oc get pods -n zero-trust-workload-identity-manager

   Example output

   NAME                    READY   STATUS    RESTARTS   AGE
   spire-agent-abcde       1/1     Running   0          10m
   spire-server-0          2/2     Running   0          10m

6. Optional: Test cross-cluster workload authentication by deploying workloads with SPIFFE identities on different clusters and verifying they can authenticate to each other using the federated trust.

Procedure

1. Install the cert-manager Operator on the cluster where you want to use externally managed certificates.

   Create a namespace and install the operator:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: cert-manager-operator
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-cert-manager-operator
     namespace: cert-manager-operator
   spec:
     upgradeStrategy: Default
   ---
   apiVersion: operators.coreos.com/stable-v1
   kind: Subscription
   metadata:
     name: openshift-cert-manager-operator
     namespace: cert-manager-operator
   spec:
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     name: openshift-cert-manager-operator
     channel: stable-v1

2. Apply the cert-manager installation by running the following command:

   $ oc apply -f cert-manager-install.yaml

3. Check the status of the cert-manager Operator by entering the following command:

   $ oc get pods -n cert-manager

   All cert-manager pods should be in Running status.

4. Create an Issuer for certificate provisioning.

   For Let’s Encrypt with HTTP-01 challenge:

   apiVersion: cert-manager.io/v1
   kind: Issuer
   metadata:
     name: letsencrypt-http01
     namespace: zero-trust-workload-identity-manager
   spec:
     acme:
       server: https://acme-v02.api.letsencrypt.org/directory
       privateKeySecretRef:
         name: letsencrypt-account-key
       solvers:
         - http01:
             ingress:
               ingressClassName: openshift-default

   Alternatively, for an internal CA:

   apiVersion: cert-manager.io/v1
   kind: Issuer
   metadata:
     name: internal-ca
     namespace: zero-trust-workload-identity-manager
   spec:
     ca:
       secretName: internal-ca-key-pair

5. Apply the Issuer by running the following command:

   $ oc apply -f issuer.yaml

6. Determine the federation endpoint domain name.

   The federation route follows a predictable naming pattern if managedRoute is set to true. Get your cluster’s application domain by running the following command:

   $ CLUSTER_DOMAIN=$(oc get ingresses.config/cluster -o jsonpath='{.spec.domain}')
   $ FEDERATION_DOMAIN="federation.${CLUSTER_DOMAIN}"
   $ echo "Federation domain will be: $FEDERATION_DOMAIN"

   Example output

   Federation domain will be: federation.apps.cluster1.example.com

   Note

   The federation route is created automatically if managedRoute is set to true when you apply the SpireServer configuration in a later step. The route name is spire-server-federation and the hostname is federation.<cluster-apps-domain>.

7. Create a Certificate resource to request a TLS certificate.

   Use the federation domain determined in the previous step:

   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: spire-server-federation-tls
     namespace: zero-trust-workload-identity-manager
   spec:
     secretName: spire-server-federation-tls
     duration: 2160h
     renewBefore: 360h
     commonName: federation.apps.cluster1.example.com
     dnsNames:
       - federation.apps.cluster1.example.com
     usages:
       - server auth
       - digital signature
       - key encipherment
     issuerRef:
       kind: Issuer
       name: letsencrypt-http01

   • The secretName field must match the externalSecretRef value in SpireServer.
   • The duration field shows how long a certificate is valid. Certificates are valid for 90 days.
   • The renewBefore field shows how many days a certificate must be renewed before it expires. Renew a certificate 15 days before expiration.
   • The commonName field must be replaced with your actual federation domain from the previous step.
   • The dnsNames field must match the commonName and the actual route hostname that was created.
   • The name field must reference the Issuer that was created earlier.

8. Apply the Certificate resource by running the following command:

   $ oc apply -f certificate.yaml

9. Monitor the certificate issuance by running the following command:

   $ oc get certificate spire-server-federation-tls \
       -n zero-trust-workload-identity-manager -w

   Example output when ready

   NAME                            READY   SECRET                          AGE
   spire-server-federation-tls     True    spire-server-federation-tls     2m

10. Create RBAC permissions for the OpenShift Ingress Router to access the certificate secret.

    Create a Role by running the following command:

    $ oc create role secret-reader \
        --verb=get,list,watch \
        --resource=secrets \
        --resource-name=spire-server-federation-tls \
        -n zero-trust-workload-identity-manager

    Create a RoleBinding by running the following command:

    $ oc create rolebinding secret-reader-binding \
        --role=secret-reader \
        --serviceaccount=openshift-ingress:router \
        -n zero-trust-workload-identity-manager

11. Configure the SpireServer custom resource to use manual certificate management.

    Now that the certificate is ready, configure the SpireServer to reference it:

    apiVersion: operator.openshift.io/v1alpha1
    kind: SpireServer
    metadata:
      name: cluster
    spec:
      trustDomain: cluster1.example.com
      federation:
        bundleEndpoint:
          profile: https_web
          refreshHint: 300
          httpsWeb:
            servingCert:
              fileSyncInterval: 86400
              externalSecretRef: spire-server-federation-tls
        managedRoute: "true"

    • The profile field must use https_web profile for certificate-based authentication.
    • The fileSyncInterval field checks for certificate updates every 24 hours (86400 seconds). Range: 3600-7776000 seconds.
    • The externalSecretRef field is the name of the secret containing the TLS certificate and private key. Must match the certificate secret created in the previous steps.

12. Apply the configuration by running the following command:

    $ oc apply -f spireserver.yaml

13. Wait for the SPIRE Server to be ready:

    $ oc get spireserver cluster -n zero-trust-workload-identity-manager -w

    Wait until the status shows Ready.

14. Verify that the federation route was created by running the following command:

    $ oc get route spire-server-federation -n zero-trust-workload-identity-manager

    Example output

    NAME                      HOST/PORT                                  PATH   SERVICES        PORT    TERMINATION
    spire-server-federation   federation.apps.cluster1.example.com              spire-server    8443    reencrypt

    Verify that the route hostname matches the domain name used in your certificate.

15. Verify that the federation endpoint is accessible by running the following command:

    $ curl https://$(oc get route spire-server-federation \
        -n zero-trust-workload-identity-manager \
        -o jsonpath='{.spec.host}')

    You should receive a JSON response containing the trust bundle.

16. Fetch the trust bundle from each federation endpoint that you want to federate with.

    For each remote cluster, fetch its trust bundle by running the following commands:

    $ curl https://federation.apps.cluster1.example.com > cluster1-bundle.json
    $ curl https://federation.apps.cluster2.example.com > cluster2-bundle.json

    The trust bundle is in JSON Web Key Set (JWKS) format:

    Example trust bundle

    {
      "keys": [
        {
          "use": "x509-svid",
          "kty": "RSA",
          "n": "xGOzB...",
          "e": "AQAB",
          "x5c": ["MIIC..."]
        }
      ],
      "spiffe_sequence": 1,
      "refresh_hint": 300
    }

17. Create ClusterFederatedTrustDomain resources for each remote trust domain you want to federate with:

    apiVersion: spire.spiffe.io/v1alpha1
    kind: ClusterFederatedTrustDomain
    metadata:
      name: cluster1-federation
    spec:
      trustDomain: cluster1.example.com
      bundleEndpointURL: https://federation.apps.cluster1.example.com
      bundleEndpointProfile:
        type: https_web
      className: zero-trust-workload-identity-manager-spire
      trustDomainBundle: |
        {
          "keys": [
            {
              "use": "x509-svid",
              "kty": "RSA",
              "n": "xGOzB...",
              "e": "AQAB",
              "x5c": ["MIIC..."]
            }
          ],
          "spiffe_sequence": 1
        }
    ---
    apiVersion: spire.spiffe.io/v1alpha1
    kind: ClusterFederatedTrustDomain
    metadata:
      name: cluster2-federation
    spec:
      trustDomain: cluster2.example.com
      bundleEndpointURL: https://federation.apps.cluster2.example.com
      bundleEndpointProfile:
        type: https_web
      className: zero-trust-workload-identity-manager-spire
      trustDomainBundle: |
        {
          "keys": [...],
          "spiffe_sequence": 1
        }

    • The trustDomainBundle field contains the complete trust bundle JSON that you fetched in the previous step.
    • The spec.className field contains the name of a class to watch CRs for. Spire-controller-manager watches the resource only if spec.className is set to zero-trust-workload-identity-manager-spire.

18. Apply the ClusterFederatedTrustDomain resources by running the following command:

    $ oc apply -f clusterfederatedtrustdomains.yaml

19. Update the SpireServer resource to add the federatesWith configuration:

    apiVersion: operator.openshift.io/v1alpha1
    kind: SpireServer
    metadata:
      name: cluster
    spec:
      trustDomain: cluster3.example.com
      federation:
        bundleEndpoint:
          profile: https_web
          refreshHint: 300
          httpsWeb:
            servingCert:
              fileSyncInterval: 86400
              externalSecretRef: spire-server-federation-tls
        federatesWith:
          - trustDomain: cluster1.example.com
            bundleEndpointUrl: https://federation.apps.cluster1.example.com
            bundleEndpointProfile: https_web
    
          - trustDomain: cluster2.example.com
            bundleEndpointUrl: https://federation.apps.cluster2.example.com
            bundleEndpointProfile: https_web
        managedRoute: "true"

    • The federatesWith field lists all remote trust domains this cluster should federate with.

20. Apply the updated configuration by running the following command:

    $ oc apply -f spireserver.yaml

21. Repeat steps 1-15 on each cluster that participates in the federation, ensuring that:

    • Each cluster has cert-manager installed and configured
    • Each cluster has its own certificate created and ready before applying the SpireServer configuration
    • Each cluster has the RBAC for the ingress router configured
    • Each cluster has ClusterFederatedTrustDomain resources for every other cluster it federates with
    • Each cluster’s SpireServer has the complete federatesWith list

Verification

1. Verify that the certificate has been issued successfully by running the following command:

   $ oc get certificate spire-server-federation-tls \
       -n zero-trust-workload-identity-manager

   Example output

   NAME                            READY   SECRET                          AGE
   spire-server-federation-tls     True    spire-server-federation-tls     5m

2. Check the certificate details and expiration by running the following command:

   $ oc get secret spire-server-federation-tls \
       -n zero-trust-workload-identity-manager \
       -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates

   Example output

   notBefore=Dec 16 10:00:00 2025 GMT
   notAfter=Mar 16 10:00:00 2026 GMT

3. Verify that the RBAC permissions are configured correctly by running the following command:

   $ oc get role,rolebinding -n zero-trust-workload-identity-manager \
       | grep secret-reader

   Example output

   role.rbac.authorization.k8s.io/secret-reader
   rolebinding.rbac.authorization.k8s.io/secret-reader-binding

   Verify the RoleBinding references the correct ServiceAccount by running the following command:

   $ oc describe rolebinding secret-reader-binding \
       -n zero-trust-workload-identity-manager

   Example output

   Name:         secret-reader-binding
   Namespace:    zero-trust-workload-identity-manager
   Role:
     Kind:  Role
     Name:  secret-reader
   Subjects:
     Kind            Name    Namespace
     ----            ----    ---------
     ServiceAccount  router  openshift-ingress

4. Verify that the ClusterFederatedTrustDomain resources have been created by running the following command:

   $ oc get clusterfederatedtrustdomains

   Example output

   NAME                  TRUST DOMAIN           ENDPOINT URL                                      AGE
   cluster1-federation   cluster1.example.com   https://federation.apps.cluster1.example.com     5m
   cluster2-federation   cluster2.example.com   https://federation.apps.cluster2.example.com     5m

5. Check the status of a ClusterFederatedTrustDomain to ensure bundle synchronization is working by running the following command:

   $ oc describe clusterfederatedtrustdomain cluster1-federation

   Look for successful status conditions indicating that the trust bundle has been synchronized.

6. Verify that the federation endpoint is accessible and using the correct certificate by running the following command:

   $ curl -v https://$(oc get route spire-server-federation \
       -n zero-trust-workload-identity-manager \
       -o jsonpath='{.spec.host}')

   In the output, verify that the certificate presented is issued by your configured CA (Let’s Encrypt or internal CA).

7. Check the SPIRE Server logs to confirm that by running the following command:

   • Federation is active with remote trust domains
   • Trust bundles are being synchronized

   • The bundle endpoint is serving correctly

     $ oc logs -n zero-trust-workload-identity-manager \
         statefulset/spire-server -c spire-server --tail=100

     Look for log messages indicating successful federation bundle synchronization.

8. Verify that all SPIRE components are running by running the following command:

   $ oc get pods -n zero-trust-workload-identity-manager

   Example output

   NAME                                    READY   STATUS    RESTARTS   AGE
   spire-agent-abc123                      1/1     Running   0          10m
   spire-server-0                          2/2     Running   0          10m

9. Optional: Test cross-cluster workload authentication by deploying workloads with SPIFFE identities on different clusters and verifying they can authenticate to each other using the federated trust.

1. Connects to the SPIFFE Workload API, usually through the SPIFFE Runtime Environment Agent socket exposed by the SPIFFE CSI driver.
2. Fetches X.509 SPIFFE Verifiable Identity Document (SVID)s, JSON Web Token (JWT) SVIDs, and JWT bundles from SPIFFE Runtime Environment.
3. Writes credentials to files under a configured directory such as cert_dir.
4. Optionally notifies a workload when X.509 authentication credentials are renewed in daemon mode.

Procedure

1. Create a Containerfile with the following contents:

   FROM registry.redhat.io/zero-trust-workload-identity-manager/spiffe-helper-rhel9:<version>
   
   USER 0
   
   RUN dnf install -y postgresql && \
       dnf clean all
   
   USER 65534

   Replace <version> with the tag that matches your Zero Trust Workload Identity Manager installation.

2. Build and push a custom SPIFFE Helper image for the PostgreSQL server by running the following command.

   $ podman build -f Containerfile \
     -t <registry>/spiffe-helper-postgresql:latest .

   $ podman push <registry>/spiffe-helper-postgresql:latest

   Replace <registry> with your container registry.

3. Create the PostgreSQL server resources by using the following example:

   $ oc apply -f - << 'EOF'
   ---
   kind: Namespace
   apiVersion: v1
   metadata:
     name: postgresql-spiffe
   ---
   kind: ServiceAccount
   apiVersion: v1
   metadata:
     name: postgresql-spiffe
     namespace: postgresql-spiffe
   ---
   apiVersion: spire.spiffe.io/v1alpha1
   kind: ClusterSPIFFEID
   metadata:
     name: postgresql-spiffe
   spec:
     className: zero-trust-workload-identity-manager-spire
     dnsNameTemplates:
       - postgresql-spiffe.postgresql-spiffe.svc
     namespaceSelector:
       matchLabels:
         kubernetes.io/metadata.name: postgresql-spiffe
     podSelector:
       matchLabels:
         app: postgresql-spiffe
     spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}'
   ---
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: postgres-config
     namespace: postgresql-spiffe
   data:
     pg_hba.conf: |
       # TYPE      DATABASE        USER            ADDRESS                 METHOD
       local       all             all                                     trust
       host        postgres        postgres        127.0.0.1/32            trust
       hostnossl   postgres        postgres        127.0.0.1/32            trust
       hostnossl   all             all             0.0.0.0/0               reject
       hostssl     all             all             0.0.0.0/0               cert
     postgresql.conf: |
       listen_addresses '*'
       ssl = on
       ssl_cert_file = '/opt/postgresql-certs/svid.pem'
       ssl_key_file = '/opt/postgresql-certs/svid.key'
       ssl_ca_file = '/opt/postgresql-certs/svid_bundle.pem'
   ---
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: postgresql-init-db
     namespace: postgresql-spiffe
   data:
     initdb.sh: |
       # Copy configuration files
       cp /opt/pg_hba/pg_hba.conf /var/lib/pgsql/data/userdata
       # Create postgresql resources
       psql -U postgres <<!!EOF
           CREATE DATABASE $SPIFFE_DATABASE;
           CREATE USER $SPIFFE_USER WITH encrypted password '$SPIFFE_PASSWORD';
           GRANT ALL privileges ON database $SPIFFE_DATABASE TO $SPIFFE_USER;
           \c $SPIFFE_DATABASE;
           CREATE TABLE test_table (
             id bigserial primary key,
             name VARCHAR(255) NOT NULL,
             text VARCHAR(255) NOT NULL
           );
           GRANT ALL privileges ON table test_table TO $SPIFFE_USER;
           GRANT ALL privileges ON sequence test_table_id_seq TO $SPIFFE_USER;
       !!EOF
   ---
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: spiffe-helper
     namespace: postgresql-spiffe
   data:
     helper.conf: |
       agent_address = "/spiffe-workload-api/spire-agent.sock"
       cmd = "/usr/bin/psql"
       cmd_args = "-U postgres -h 127.0.0.1 -c \"SELECT pg_reload_conf();\""
       cert_dir = "/opt/postgresql-certs"
       renew_signal = ""
       svid_file_name = "svid.pem"
       svid_key_file_name = "svid.key"
       svid_bundle_file_name = "svid_bundle.pem"
   ---
   kind: Service
   apiVersion: v1
   metadata:
     name: postgresql-spiffe
     namespace: postgresql-spiffe
   spec:
     ports:
       - protocol: TCP
         port: 5432
         targetPort: 5432
     type: ClusterIP
     selector:
       app: postgresql-spiffe
   ---
   kind: Deployment
   apiVersion: apps/v1
   metadata:
     name: postgresql-spiffe
     namespace: postgresql-spiffe
     labels:
       app: postgresql-spiffe
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: postgresql-spiffe
     template:
       metadata:
         labels:
           app: postgresql-spiffe
       spec:
         serviceAccountName: postgresql-spiffe
         securityContext:
           runAsNonRoot: true
           seccompProfile:
             type: RuntimeDefault
         initContainers:
           - name: spiffe-helper-init
             image: <registry>/spiffe-helper-postgresql:latest
             args:
               - '-config'
               - /etc/spiffe-helper/helper.conf
               - '-daemon-mode=false'
             volumeMounts:
               - name: spiffe-workload-api
                 readOnly: true
                 mountPath: /spiffe-workload-api
               - name: postgresql-certs
                 mountPath: /opt/postgresql-certs
               - name: spiffe-helper
                 mountPath: /etc/spiffe-helper
             securityContext:
               capabilities:
                 drop:
                   - ALL
         containers:
           - name: postgresql-spiffe
             image: registry.redhat.io/rhel9/postgresql-16:latest
             env:
               - name: POSTGRESQL_USER
                 value: user
               - name: POSTGRESQL_PASSWORD
                 value: wearenotusingthissoIdontworryaboutit
               - name: SPIFFE_USER
                 value: postgresql_spiffe
               - name: SPIFFE_PASSWORD
                 value: somealsonotusedpassword
               - name: SPIFFE_DATABASE
                 value: testdb
               - name: POSTGRESQL_DATABASE
                 value: sampledb
             ports:
               - name: postgresql
                 containerPort: 5432
                 protocol: TCP
             volumeMounts:
               - name: postgresql-certs
                 mountPath: /opt/postgresql-certs
               - name: pg-hba
                 mountPath: /opt/pg_hba
               - name: postgresql-init-db
                 mountPath: /opt/app-root/src/postgresql-init
               - name: postgres-config
                 mountPath: /opt/app-root/src/postgresql-cfg
             securityContext:
               capabilities:
                 drop:
                   - ALL
           - name: spiffe-helper
             image: <registry>/spiffe-helper-postgresql:latest
             args:
               - '-config'
               - /etc/spiffe-helper/helper.conf
             volumeMounts:
               - name: spiffe-workload-api
                 readOnly: true
                 mountPath: /spiffe-workload-api
               - name: postgresql-certs
                 mountPath: /opt/postgresql-certs
               - name: spiffe-helper
                 mountPath: /etc/spiffe-helper
             securityContext:
               capabilities:
                 drop:
                   - ALL
         volumes:
           - name: spiffe-workload-api
             csi:
               driver: csi.spiffe.io
               readOnly: true
           - name: postgresql-certs
             emptyDir:
               medium: Memory
           - name: spiffe-helper
             configMap:
               name: spiffe-helper
           - name: postgresql-init-db
             configMap:
               name: postgresql-init-db
           - name: pg-hba
             configMap:
               name: postgres-config
               items:
                 - key: pg_hba.conf
                   path: pg_hba.conf
           - name: postgres-config
             configMap:
               name: postgres-config
               items:
                 - key: postgresql.conf
                   path: postgresql.conf
   EOF

   Replace both <registry> occurrences in the deployment with your container registry.

4. Confirm that the SPIFFE Helper configuration matches the PostgreSQL TLS settings by running the following command:

   $ oc get configmap spiffe-helper -n postgresql-spiffe \
     -o jsonpath='{.data.helper\.conf}{"\n"}'

   The cert_dir and svid_* file names must match the ssl_*_file paths in postgresql.conf. The server helper.conf must include settings similar to the following example:

   agent_address = "/spiffe-workload-api/spire-agent.sock"
   cmd = "/usr/bin/psql"
   cmd_args = "-U postgres -h 127.0.0.1 -c \"SELECT pg_reload_conf();\""
   cert_dir = "/opt/postgresql-certs"
   renew_signal = ""
   svid_file_name = "svid.pem"
   svid_key_file_name = "svid.key"
   svid_bundle_file_name = "svid_bundle.pem"

5. Create the PostgreSQL client by using the following example. The client uses the stock SPIFFE Helper image from the Zero Trust Workload Identity Manager. Only the server deployment requires the custom image with the psql client.

   $ oc apply -f - << 'EOF'
   ---
   kind: Namespace
   apiVersion: v1
   metadata:
     name: postgresql-spiffe-client
   ---
   kind: ServiceAccount
   apiVersion: v1
   metadata:
     name: postgresql-spiffe-client
     namespace: postgresql-spiffe-client
   ---
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: spiffe-helper
     namespace: postgresql-spiffe-client
   data:
     helper.conf: |
       agent_address = "/spiffe-workload-api/spire-agent.sock"
       cmd = ""
       cmd_args = ""
       cert_dir = "/opt/postgresql-certs"
       renew_signal = ""
       svid_file_name = "svid.pem"
       svid_key_file_name = "svid.key"
       svid_bundle_file_name = "svid_bundle.pem"
   ---
   apiVersion: spire.spiffe.io/v1alpha1
   kind: ClusterSPIFFEID
   metadata:
     name: postgresql-spiffe-client
   spec:
     className: zero-trust-workload-identity-manager-spire
     dnsNameTemplates:
       - postgresql-spiffe
     namespaceSelector:
       matchLabels:
         kubernetes.io/metadata.name: postgresql-spiffe-client
     podSelector:
       matchLabels:
         app: postgresql-spiffe-client
     spiffeIDTemplate: 'spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}'
   ---
   kind: Deployment
   apiVersion: apps/v1
   metadata:
     name: postgresql-spiffe-client
     namespace: postgresql-spiffe-client
     labels:
       app: postgresql-spiffe-client
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: postgresql-spiffe-client
     template:
       metadata:
         labels:
           app: postgresql-spiffe-client
       spec:
         serviceAccountName: postgresql-spiffe-client
         securityContext:
           runAsNonRoot: true
           seccompProfile:
             type: RuntimeDefault
         containers:
           - name: postgresql-spiffe-client
             image: registry.redhat.io/rhel9/postgresql-16:latest
             command:
               - /bin/bash
               - '-c'
               - sleep infinity
             volumeMounts:
               - name: postgresql-certs
                 mountPath: /opt/postgresql-certs
             securityContext:
               capabilities:
                 drop:
                   - ALL
           - name: spiffe-helper
             image: registry.redhat.io/zero-trust-workload-identity-manager/spiffe-helper-rhel9:
             args:
               - '-config'
               - /etc/spiffe-helper/helper.conf
             volumeMounts:
               - name: spiffe-workload-api
                 readOnly: true
                 mountPath: /spiffe-workload-api
               - name: postgresql-certs
                 mountPath: /opt/postgresql-certs
               - name: spiffe-helper
                 mountPath: /etc/spiffe-helper
             securityContext:
               capabilities:
                 drop:
                   - ALL
         volumes:
           - name: spiffe-workload-api
             csi:
               driver: csi.spiffe.io
               readOnly: true
           - name: postgresql-certs
             emptyDir:
               medium: Memory
           - name: spiffe-helper
             configMap:
               name: spiffe-helper
   EOF

   Replace <version> with the tag that matches your Zero Trust Workload Identity Manager installation.

6. Confirm that pods are running in both namespaces by running the following commands:

   $ oc get pods -n postgresql-spiffe

   $ oc get pods -n postgresql-spiffe-client