Chapter 9. cert-manager Operator for Red Hat OpenShift

Procedure

1. Create a config map in the cert-manager namespace by running the following command:

   $ oc create configmap trusted-ca -n cert-manager

2. Inject the CA bundle that is trusted by OpenShift Container Platform into the config map by running the following command:

   $ oc label cm trusted-ca config.openshift.io/inject-trusted-cabundle=true -n cert-manager

3. Update the deployment for the cert-manager Operator for Red Hat OpenShift to use the config map by running the following command:

   $ oc -n cert-manager-operator patch subscription openshift-cert-manager-operator --type='merge' -p '{"spec":{"config":{"env":[{"name":"TRUSTED_CA_CONFIGMAP_NAME","value":"trusted-ca"}]}}}'

Verification

1. Verify that the deployments have finished rolling out by running the following command:

   $ oc rollout status deployment/cert-manager-operator-controller-manager -n cert-manager-operator && \
   oc rollout status deployment/cert-manager -n cert-manager && \
   oc rollout status deployment/cert-manager-webhook -n cert-manager && \
   oc rollout status deployment/cert-manager-cainjector -n cert-manager

   Example output

   deployment "cert-manager-operator-controller-manager" successfully rolled out
   deployment "cert-manager" successfully rolled out
   deployment "cert-manager-webhook" successfully rolled out
   deployment "cert-manager-cainjector" successfully rolled out

2. Verify that the CA bundle was mounted as a volume by running the following command:

   $ oc get deployment cert-manager -n cert-manager -o=jsonpath={.spec.template.spec.'containers[0].volumeMounts'}

   Example output

   [{"mountPath":"/etc/pki/tls/certs/cert-manager-tls-ca-bundle.crt","name":"trusted-ca","subPath":"ca-bundle.crt"}]

3. Verify that the source of the CA bundle is the trusted-ca config map by running the following command:

   $ oc get deployment cert-manager -n cert-manager -o=jsonpath={.spec.template.spec.volumes}

   Example output

   [{"configMap":{"defaultMode":420,"name":"trusted-ca"},"name":"trusted-ca"}]

Procedure

1. Edit the CertManager resource by running the following command:

   $ oc edit certmanager cluster

2. Add a spec.controllerConfig section with the following override arguments:

   apiVersion: operator.openshift.io/v1alpha1
   kind: CertManager
   metadata:
     name: cluster
     ...
   spec:
     ...
     controllerConfig:
       overrideEnv:
         - name: HTTP_PROXY
           value: http://<proxy_url> 1
         - name: HTTPS_PROXY
           value: https://<proxy_url> 2
         - name: NO_PROXY
           value: <ignore_proxy_domains> 3

   1 2

   Replace <proxy_url> with the proxy server URL.

   3

   Replace <ignore_proxy_domains> with a comma separated list of domains. These domains are ignored by the proxy server.

3. Save your changes and quit the text editor to apply your changes.

Verification

1. Verify that the cert-manager controller pod is redeployed by running the following command:

   $ oc get pods -l app.kubernetes.io/name=cert-manager -n cert-manager

   Example output

   NAME                          READY   STATUS    RESTARTS   AGE
   cert-manager-bd7fbb9fc-wvbbt  1/1     Running   0          39s

2. Verify that environment variables are updated for the cert-manager pod by running the following command:

   $ oc get pod <redeployed_cert-manager_controller_pod> -n cert-manager -o yaml

   Example output

       env:
       ...
       - name: HTTP_PROXY
         value: http://<PROXY_URL>
       - name: HTTPS_PROXY
         value: https://<PROXY_URL>
       - name: NO_PROXY
         value: <IGNORE_PROXY_DOMAINS>

Procedure

1. Edit the CertManager resource by running the following command:

   $ oc edit certmanager cluster

2. Add a spec.controllerConfig section with the following override arguments:

   apiVersion: operator.openshift.io/v1alpha1
   kind: CertManager
   metadata:
     name: cluster
     ...
   spec:
     ...
     controllerConfig:
       overrideArgs:
         - '--dns01-recursive-nameservers=<server_address>' 1
         - '--dns01-recursive-nameservers-only' 2
         - '--acme-http01-solver-nameservers=<host>:<port>' 3
         - '--v=<verbosity_level>' 4
         - '--metrics-listen-address=<host>:<port>' 5
         - '--issuer-ambient-credentials' 6
     webhookConfig:
       overrideArgs:
         - '--v=4' 7
     cainjectorConfig:
       overrideArgs:
         - '--v=2' 8

   1

   Provide a comma-separated list of nameservers to query for the DNS-01 self check. The nameservers can be specified either as <host>:<port>, for example, 1.1.1.1:53, or use DNS over HTTPS (DoH), for example, https://1.1.1.1/dns-query.

   2

   Specify to only use recursive nameservers instead of checking the authoritative nameservers associated with that domain.

   3

   Provide a comma-separated list of <host>:<port> nameservers to query for the Automated Certificate Management Environment (ACME) HTTP01 self check. For example, --acme-http01-solver-nameservers=1.1.1.1:53.

   4 7 8

   Specify to set the log level verbosity to determine the verbosity of log messages.

   5

   Specify the host and port for the metrics endpoint. The default value is --metrics-listen-address=0.0.0.0:9402.

   6

   You must use the --issuer-ambient-credentials argument when configuring an ACME Issuer to solve DNS-01 challenges by using ambient credentials.

   Note

   DNS over HTTPS (DoH) is supported starting only from cert-manager Operator for Red Hat OpenShift version 1.13.0 and later.

3. Save your changes and quit the text editor to apply your changes.

Procedure

1. Check that the Certificate object and its secret are available by running the following command:

   $ oc get certificate

   Example output

   NAME                                             READY   SECRET                                           AGE
   certificate-from-clusterissuer-route53-ambient   True    certificate-from-clusterissuer-route53-ambient   8h

2. Edit the CertManager resource by running the following command:

   $ oc edit certmanager cluster

3. Add a spec.controllerConfig section with the following override arguments:

   apiVersion: operator.openshift.io/v1alpha1
   kind: CertManager
   metadata:
     name: cluster
   # ...
   spec:
   # ...
     controllerConfig:
       overrideArgs:
         - '--enable-certificate-owner-ref'

4. Save your changes and quit the text editor to apply your changes.

Procedure

1. Check that the deployments of the cert-manager controller, CA injector, and Webhook are available by entering the following command:

   $ oc get deployment -n cert-manager

   Example output

   NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
   cert-manager              1/1     1            1           53m
   cert-manager-cainjector   1/1     1            1           53m
   cert-manager-webhook      1/1     1            1           53m

2. Before setting the CPU and memory limit, check the existing configuration for the cert-manager controller, CA injector, and Webhook by entering the following command:

   $ oc get deployment -n cert-manager -o yaml

   Example output

   # ...
     metadata:
       name: cert-manager
       namespace: cert-manager
   # ...
     spec:
       template:
         spec:
           containers:
           - name: cert-manager-controller
             resources: {} 1
   # ...
     metadata:
       name: cert-manager-cainjector
       namespace: cert-manager
   # ...
     spec:
       template:
         spec:
           containers:
           - name: cert-manager-cainjector
             resources: {} 2
   # ...
     metadata:
       name: cert-manager-webhook
       namespace: cert-manager
   # ...
     spec:
       template:
         spec:
           containers:
           - name: cert-manager-webhook
             resources: {} 3
   # ...

   1 2 3

   The spec.resources field is empty by default. The cert-manager components do not have CPU and memory limits.

3. To configure the CPU and memory limits for the cert-manager controller, CA injector, and Webhook, enter the following command:

   $ oc patch certmanager.operator cluster --type=merge -p="
   spec:
     controllerConfig:
       overrideResources:
         limits: 1
           cpu: 200m 2
           memory: 64Mi 3
         requests: 4
           cpu: 10m 5
           memory: 16Mi 6
     webhookConfig:
       overrideResources:
         limits: 7
           cpu: 200m 8
           memory: 64Mi 9
         requests: 10
           cpu: 10m 11
           memory: 16Mi 12
     cainjectorConfig:
       overrideResources:
         limits: 13
           cpu: 200m 14
           memory: 64Mi 15
         requests: 16
           cpu: 10m 17
           memory: 16Mi 18
   "

   1

   Defines the maximum amount of CPU and memory that a single container in a cert-manager controller pod can request.

   2 5

   You can specify the CPU limit that a cert-manager controller pod can request. The default value is 10m.

   3 6

   You can specify the memory limit that a cert-manager controller pod can request. The default value is 32Mi.

   4

   Defines the amount of CPU and memory set by scheduler for the cert-manager controller pod.

   7

   Defines the maximum amount of CPU and memory that a single container in a CA injector pod can request.

   8 11

   You can specify the CPU limit that a CA injector pod can request. The default value is 10m.

   9 12

   You can specify the memory limit that a CA injector pod can request. The default value is 32Mi.

   10

   Defines the amount of CPU and memory set by scheduler for the CA injector pod.

   13

   Defines the maximum amount of CPU and memory Defines the maximum amount of CPU and memory that a single container in a Webhook pod can request.

   14 17

   You can specify the CPU limit that a Webhook pod can request. The default value is 10m.

   15 18

   You can specify the memory limit that a Webhook pod can request. The default value is 32Mi.

   16

   Defines the amount of CPU and memory set by scheduler for the Webhook pod.

   Example output

   certmanager.operator.openshift.io/cluster patched

Verification

1. Verify that the CPU and memory limits are updated for the cert-manager components:

   $ oc get deployment -n cert-manager -o yaml

   Example output

   # ...
     metadata:
       name: cert-manager
       namespace: cert-manager
   # ...
     spec:
       template:
         spec:
           containers:
           - name: cert-manager-controller
             resources:
               limits:
                 cpu: 200m
                 memory: 64Mi
               requests:
                 cpu: 10m
                 memory: 16Mi
   # ...
     metadata:
       name: cert-manager-cainjector
       namespace: cert-manager
   # ...
     spec:
       template:
         spec:
           containers:
           - name: cert-manager-cainjector
             resources:
               limits:
                 cpu: 200m
                 memory: 64Mi
               requests:
                 cpu: 10m
                 memory: 16Mi
   # ...
     metadata:
       name: cert-manager-webhook
       namespace: cert-manager
   # ...
     spec:
       template:
         spec:
           containers:
           - name: cert-manager-webhook
             resources:
               limits:
                 cpu: 200m
                 memory: 64Mi
               requests:
                 cpu: 10m
                 memory: 16Mi
   # ...

Procedure

1. Create a CredentialsRequest resource YAML file, for example, sample-credential-request.yaml, as follows:

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: cert-manager
     namespace: openshift-cloud-credential-operator
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AWSProviderSpec
       statementEntries:
       - action:
         - "route53:GetChange"
         effect: Allow
         resource: "arn:aws:route53:::change/*"
       - action:
         - "route53:ChangeResourceRecordSets"
         - "route53:ListResourceRecordSets"
         effect: Allow
         resource: "arn:aws:route53:::hostedzone/*"
       - action:
         - "route53:ListHostedZonesByName"
         effect: Allow
         resource: "*"
     secretRef:
       name: aws-creds
       namespace: cert-manager
     serviceAccountNames:
     - cert-manager

2. Create a CredentialsRequest resource by running the following command:

   $ oc create -f sample-credential-request.yaml

3. Update the subscription object for cert-manager Operator for Red Hat OpenShift by running the following command:

   $ oc -n cert-manager-operator patch subscription openshift-cert-manager-operator --type=merge -p '{"spec":{"config":{"env":[{"name":"CLOUD_CREDENTIALS_SECRET_NAME","value":"aws-creds"}]}}}'

Verification

1. Get the name of the redeployed cert-manager controller pod by running the following command:

   $ oc get pods -l app.kubernetes.io/name=cert-manager -n cert-manager

   Example output

   NAME                          READY   STATUS    RESTARTS   AGE
   cert-manager-bd7fbb9fc-wvbbt  1/1     Running   0          15m39s

2. Verify that the cert-manager controller pod is updated with AWS credential volumes that are mounted under the path specified in mountPath by running the following command:

   $ oc get -n cert-manager pod/<cert-manager_controller_pod_name> -o yaml

   Example output

   ...
   spec:
     containers:
     - args:
       ...
       - mountPath: /.aws
         name: cloud-credentials
     ...
     volumes:
     ...
     - name: cloud-credentials
       secret:
         ...
         secretName: aws-creds

Procedure

1. Create a directory to store a CredentialsRequest resource YAML file by running the following command:

   $ mkdir credentials-request

2. Create a CredentialsRequest resource YAML file under the credentials-request directory, such as, sample-credential-request.yaml, by applying the following yaml:

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: cert-manager
     namespace: openshift-cloud-credential-operator
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: AWSProviderSpec
       statementEntries:
       - action:
         - "route53:GetChange"
         effect: Allow
         resource: "arn:aws:route53:::change/*"
       - action:
         - "route53:ChangeResourceRecordSets"
         - "route53:ListResourceRecordSets"
         effect: Allow
         resource: "arn:aws:route53:::hostedzone/*"
       - action:
         - "route53:ListHostedZonesByName"
         effect: Allow
         resource: "*"
     secretRef:
       name: aws-creds
       namespace: cert-manager
     serviceAccountNames:
     - cert-manager

3. Use the ccoctl tool to process CredentialsRequest objects by running the following command:

   $ ccoctl aws create-iam-roles \
       --name <user_defined_name> --region=<aws_region> \
       --credentials-requests-dir=<path_to_credrequests_dir> \
       --identity-provider-arn <oidc_provider_arn> --output-dir=<path_to_output_dir>

   Example output

   2023/05/15 18:10:34 Role arn:aws:iam::XXXXXXXXXXXX:role/<user_defined_name>-cert-manager-aws-creds created
   2023/05/15 18:10:34 Saved credentials configuration to: <path_to_output_dir>/manifests/cert-manager-aws-creds-credentials.yaml
   2023/05/15 18:10:35 Updated Role policy for Role <user_defined_name>-cert-manager-aws-creds

   Copy the <aws_role_arn> from the output to use in the next step. For example, "arn:aws:iam::XXXXXXXXXXXX:role/<user_defined_name>-cert-manager-aws-creds"

4. Add the eks.amazonaws.com/role-arn="<aws_role_arn>" annotation to the service account by running the following command:

   $ oc -n cert-manager annotate serviceaccount cert-manager eks.amazonaws.com/role-arn="<aws_role_arn>"

5. To create a new pod, delete the existing cert-manager controller pod by running the following command:

   $ oc delete pods -l app.kubernetes.io/name=cert-manager -n cert-manager

   The AWS credentials are applied to a new cert-manager controller pod within a minute.

Verification

1. Get the name of the updated cert-manager controller pod by running the following command:

   $ oc get pods -l app.kubernetes.io/name=cert-manager -n cert-manager

   Example output

   NAME                          READY   STATUS    RESTARTS   AGE
   cert-manager-bd7fbb9fc-wvbbt  1/1     Running   0          39s

2. Verify that AWS credentials are updated by running the following command:

   $ oc set env -n cert-manager po/<cert_manager_controller_pod_name> --list

   Example output

   # pods/cert-manager-57f9555c54-vbcpg, container cert-manager-controller
   # POD_NAMESPACE from field path metadata.namespace
   AWS_ROLE_ARN=XXXXXXXXXXXX
   AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Procedure

1. Create a CredentialsRequest resource YAML file, such as, sample-credential-request.yaml by applying the following yaml:

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: cert-manager
     namespace: openshift-cloud-credential-operator
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: GCPProviderSpec
       predefinedRoles:
       - roles/dns.admin
     secretRef:
       name: gcp-credentials
       namespace: cert-manager
     serviceAccountNames:
     - cert-manager

   Note

   The dns.admin role provides admin privileges to the service account for managing Google Cloud DNS resources. To ensure that the cert-manager runs with the service account that has the least privilege, you can create a custom role with the following permissions:

   • dns.resourceRecordSets.*
   • dns.changes.*
   • dns.managedZones.list

2. Create a CredentialsRequest resource by running the following command:

   $ oc create -f sample-credential-request.yaml

3. Update the subscription object for cert-manager Operator for Red Hat OpenShift by running the following command:

   $ oc -n cert-manager-operator patch subscription openshift-cert-manager-operator --type=merge -p '{"spec":{"config":{"env":[{"name":"CLOUD_CREDENTIALS_SECRET_NAME","value":"gcp-credentials"}]}}}'

Verification

1. Get the name of the redeployed cert-manager controller pod by running the following command:

   $ oc get pods -l app.kubernetes.io/name=cert-manager -n cert-manager

   Example output

   NAME                                       READY   STATUS    RESTARTS   AGE
   cert-manager-bd7fbb9fc-wvbbt               1/1     Running   0          15m39s

2. Verify that the cert-manager controller pod is updated with GCP credential volumes that are mounted under the path specified in mountPath by running the following command:

   $ oc get -n cert-manager pod/<cert-manager_controller_pod_name> -o yaml

   Example output

   spec:
     containers:
     - args:
       ...
       volumeMounts:
       ...
       - mountPath: /.config/gcloud
         name: cloud-credentials
       ....
     volumes:
     ...
     - name: cloud-credentials
       secret:
         ...
         items:
         - key: service_account.json
           path: application_default_credentials.json
         secretName: gcp-credentials

Procedure

1. Create a directory to store a CredentialsRequest resource YAML file by running the following command:

   $ mkdir credentials-request

2. In the credentials-request directory, create a YAML file that contains the following CredentialsRequest manifest:

   apiVersion: cloudcredential.openshift.io/v1
   kind: CredentialsRequest
   metadata:
     name: cert-manager
     namespace: openshift-cloud-credential-operator
   spec:
     providerSpec:
       apiVersion: cloudcredential.openshift.io/v1
       kind: GCPProviderSpec
       predefinedRoles:
       - roles/dns.admin
     secretRef:
       name: gcp-credentials
       namespace: cert-manager
     serviceAccountNames:
     - cert-manager

   Note

   The dns.admin role provides admin privileges to the service account for managing Google Cloud DNS resources. To ensure that the cert-manager runs with the service account that has the least privilege, you can create a custom role with the following permissions:

   • dns.resourceRecordSets.*
   • dns.changes.*
   • dns.managedZones.list

3. Use the ccoctl tool to process CredentialsRequest objects by running the following command:

   $ ccoctl gcp create-service-accounts \
       --name <user_defined_name> --output-dir=<path_to_output_dir> \
       --credentials-requests-dir=<path_to_credrequests_dir> \
       --workload-identity-pool <workload_identity_pool> \
       --workload-identity-provider <workload_identity_provider> \
       --project <gcp_project_id>

   Example command

   $ ccoctl gcp create-service-accounts \
       --name abcde-20230525-4bac2781 --output-dir=/home/outputdir \
       --credentials-requests-dir=/home/credentials-requests \
       --workload-identity-pool abcde-20230525-4bac2781 \
       --workload-identity-provider abcde-20230525-4bac2781 \
       --project openshift-gcp-devel

4. Apply the secrets generated in the manifests directory of your cluster by running the following command:

   $ ls <path_to_output_dir>/manifests/*-credentials.yaml | xargs -I{} oc apply -f {}

5. Update the subscription object for cert-manager Operator for Red Hat OpenShift by running the following command:

   $ oc -n cert-manager-operator patch subscription openshift-cert-manager-operator --type=merge -p '{"spec":{"config":{"env":[{"name":"CLOUD_CREDENTIALS_SECRET_NAME","value":"gcp-credentials"}]}}}'

Verification

1. Get the name of the redeployed cert-manager controller pod by running the following command:

   $ oc get pods -l app.kubernetes.io/name=cert-manager -n cert-manager

   Example output

   NAME                          READY   STATUS    RESTARTS   AGE
   cert-manager-bd7fbb9fc-wvbbt  1/1     Running   0          15m39s

2. Verify that the cert-manager controller pod is updated with GCP workload identity credential volumes that are mounted under the path specified in mountPath by running the following command:

   $ oc get -n cert-manager pod/<cert-manager_controller_pod_name> -o yaml

   Example output

   spec:
     containers:
     - args:
       ...
       volumeMounts:
       - mountPath: /var/run/secrets/openshift/serviceaccount
         name: bound-sa-token
         ...
       - mountPath: /.config/gcloud
         name: cloud-credentials
     ...
     volumes:
     - name: bound-sa-token
       projected:
         ...
         sources:
         - serviceAccountToken:
             audience: openshift
             ...
             path: token
     - name: cloud-credentials
       secret:
         ...
         items:
         - key: service_account.json
           path: application_default_credentials.json
         secretName: gcp-credentials

Procedure

1. Create an ACME cluster issuer.

   1. Create a YAML file that defines the ClusterIssuer object:

      Example acme-cluster-issuer.yaml file

      apiVersion: cert-manager.io/v1
      kind: ClusterIssuer
      metadata:
        name: letsencrypt-staging                                        1
      spec:
        acme:
          preferredChain: ""
          privateKeySecretRef:
            name: <secret_for_private_key>                               2
          server: https://acme-staging-v02.api.letsencrypt.org/directory 3
          solvers:
          - http01:
              ingress:
                ingressClassName: openshift-default                                 4

      1

      Provide a name for the cluster issuer.

      2

      Replace <secret_private_key> with the name of secret to store the ACME account private key in.

      3

      Specify the URL to access the ACME server’s directory endpoint. This example uses the Let’s Encrypt staging environment.

      4

      Specify the Ingress class.

   2. Optional: If you create the object without specifying ingressClassName, use the following command to patch the existing ingress:

      $ oc patch ingress/<ingress-name> --type=merge --patch '{"spec":{"ingressClassName":"openshift-default"}}' -n <namespace>

   3. Create the ClusterIssuer object by running the following command:

      $ oc create -f acme-cluster-issuer.yaml

2. Create an Ingress to expose the service of the user workload.

   1. Create a YAML file that defines a Namespace object:

      Example namespace.yaml file

      apiVersion: v1
      kind: Namespace
      metadata:
        name: my-ingress-namespace 1

      1

      Specify the namespace for the Ingress.

   2. Create the Namespace object by running the following command:

      $ oc create -f namespace.yaml

   3. Create a YAML file that defines the Ingress object:

      Example ingress.yaml file

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: sample-ingress                                           1
        namespace: my-ingress-namespace                                2
        annotations:
          cert-manager.io/cluster-issuer: letsencrypt-staging          3
          acme.cert-manager.io/http01-ingress-class: openshift-default 4
      spec:
        ingressClassName: openshift-default                            5
        tls:
        - hosts:
          - <hostname>                                                 6
          secretName: sample-tls                                       7
        rules:
        - host: <hostname>                                             8
          http:
            paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: sample-workload                                9
                  port:
                    number: 80

      1

      Specify the name of the Ingress.

      2

      Specify the namespace that you created for the Ingress.

      3

      Specify the cluster issuer that you created.

      4

      Specify the Ingress class.

      5

      Specify the Ingress class.

      6

      Replace <hostname> with the Subject Alternative Name to be associated with the certificate. This name is used to add DNS names to the certificate.

      7

      Specify the secret to store the created certificate in.

      8

      Replace <hostname> with the hostname. You can use the <host_name>.<cluster_ingress_domain> syntax to take advantage of the *.<cluster_ingress_domain> wildcard DNS record and serving certificate for the cluster. For example, you might use apps.<cluster_base_domain>. Otherwise, you must ensure that a DNS record exists for the chosen hostname.

      9

      Specify the name of the service to expose. This example uses a service named sample-workload.

   4. Create the Ingress object by running the following command:

      $ oc create -f ingress.yaml

Procedure

1. Optional: Override the nameserver settings for the DNS-01 self check.

   This step is required only when the target public-hosted zone overlaps with the cluster’s default private-hosted zone.

   1. Edit the CertManager resource by running the following command:

      $ oc edit certmanager cluster

   2. Add a spec.controllerConfig section with the following override arguments:

      apiVersion: operator.openshift.io/v1alpha1
      kind: CertManager
      metadata:
        name: cluster
        ...
      spec:
        ...
        controllerConfig:                                1
          overrideArgs:
            - '--dns01-recursive-nameservers-only'       2
            - '--dns01-recursive-nameservers=1.1.1.1:53' 3

      1

      Add the spec.controllerConfig section.

      2

      Specify to only use recursive nameservers instead of checking the authoritative nameservers associated with that domain.

      3

      Provide a comma-separated list of <host>:<port> nameservers to query for the DNS-01 self check. You must use a 1.1.1.1:53 value to avoid the public and private zones overlapping.

   3. Save the file to apply the changes.

2. Optional: Create a namespace for the issuer:

   $ oc new-project <issuer_namespace>

3. Create a secret to store your AWS credentials in by running the following command:

   $ oc create secret generic aws-secret --from-literal=awsSecretAccessKey=<aws_secret_access_key> \ 1
       -n my-issuer-namespace

   1

   Replace <aws_secret_access_key> with your AWS secret access key.

4. Create an issuer:

   1. Create a YAML file that defines the Issuer object:

      Example issuer.yaml file

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: <letsencrypt_staging>                                        1
        namespace: <issuer_namespace>                                   2
      spec:
        acme:
          server: https://acme-staging-v02.api.letsencrypt.org/directory 3
          email: "<email_address>"                                       4
          privateKeySecretRef:
            name: <secret_private_key>                                   5
          solvers:
          - dns01:
              route53:
                accessKeyID: <aws_key_id>                                6
                hostedZoneID: <hosted_zone_id>                           7
                region: <region_name>                                    8
                secretAccessKeySecretRef:
                  name: "aws-secret"                                     9
                  key: "awsSecretAccessKey"                              10

      1

      Provide a name for the issuer.

      2

      Specify the namespace that you created for the issuer.

      3

      Specify the URL to access the ACME server’s directory endpoint. This example uses the Let’s Encrypt staging environment.

      4

      Replace <email_address> with your email address.

      5

      Replace <secret_private_key> with the name of the secret to store the ACME account private key in.

      6

      Replace <aws_key_id> with your AWS key ID.

      7

      Replace <hosted_zone_id> with your hosted zone ID.

      8

      Replace <region_name> with the AWS region name. For example, us-east-1.

      9

      Specify the name of the secret you created.

      10

      Specify the key in the secret you created that stores your AWS secret access key.

   2. Create the Issuer object by running the following command:

      $ oc create -f issuer.yaml

Procedure

1. Optional: Override the nameserver settings for the DNS-01 self check.

   This step is required only when the target public-hosted zone overlaps with the cluster’s default private-hosted zone.

   1. Edit the CertManager resource by running the following command:

      $ oc edit certmanager cluster

   2. Add a spec.controllerConfig section with the following override arguments:

      apiVersion: operator.openshift.io/v1alpha1
      kind: CertManager
      metadata:
        name: cluster
        ...
      spec:
        ...
        controllerConfig:                                1
          overrideArgs:
            - '--dns01-recursive-nameservers-only'       2
            - '--dns01-recursive-nameservers=1.1.1.1:53' 3

      1

      Add the spec.controllerConfig section.

      2

      Specify to only use recursive nameservers instead of checking the authoritative nameservers associated with that domain.

      3

      Provide a comma-separated list of <host>:<port> nameservers to query for the DNS-01 self check. You must use a 1.1.1.1:53 value to avoid the public and private zones overlapping.

   3. Save the file to apply the changes.

2. Optional: Create a namespace for the issuer:

   $ oc new-project <issuer_namespace>

3. Modify the CertManager resource to add the --issuer-ambient-credentials argument:

   $ oc patch certmanager/cluster \
     --type=merge \
     -p='{"spec":{"controllerConfig":{"overrideArgs":["--issuer-ambient-credentials"]}}}'

4. Create an issuer:

   1. Create a YAML file that defines the Issuer object:

      Example issuer.yaml file

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: <letsencrypt_staging>                                        1
        namespace: <issuer_namespace>                                   2
      spec:
        acme:
          server: https://acme-staging-v02.api.letsencrypt.org/directory 3
          email: "<email_address>"                                       4
          privateKeySecretRef:
            name: <secret_private_key>                                   5
          solvers:
          - dns01:
              route53:
                hostedZoneID: <hosted_zone_id>                           6
                region: us-east-1

      1

      Provide a name for the issuer.

      2

      Specify the namespace that you created for the issuer.

      3

      Specify the URL to access the ACME server’s directory endpoint. This example uses the Let’s Encrypt staging environment.

      4

      Replace <email_address> with your email address.

      5

      Replace <secret_private_key> with the name of the secret to store the ACME account private key in.

      6

      Replace <hosted_zone_id> with your hosted zone ID.

   2. Create the Issuer object by running the following command:

      $ oc create -f issuer.yaml

Procedure

1. Optional: Override the nameserver settings for the DNS-01 self check.

   This step is required only when the target public-hosted zone overlaps with the cluster’s default private-hosted zone.

   1. Edit the CertManager resource by running the following command:

      $ oc edit certmanager cluster

   2. Add a spec.controllerConfig section with the following override arguments:

      apiVersion: operator.openshift.io/v1alpha1
      kind: CertManager
      metadata:
        name: cluster
        ...
      spec:
        ...
        controllerConfig:                                1
          overrideArgs:
            - '--dns01-recursive-nameservers-only'       2
            - '--dns01-recursive-nameservers=1.1.1.1:53' 3

      1

      Add the spec.controllerConfig section.

      2

      Specify to only use recursive nameservers instead of checking the authoritative nameservers associated with that domain.

      3

      Provide a comma-separated list of <host>:<port> nameservers to query for the DNS-01 self check. You must use a 1.1.1.1:53 value to avoid the public and private zones overlapping.

   3. Save the file to apply the changes.

2. Optional: Create a namespace for the issuer:

   $ oc new-project my-issuer-namespace

3. Create a secret to store your GCP credentials by running the following command:

   $ oc create secret generic clouddns-dns01-solver-svc-acct --from-file=service_account.json=<path/to/gcp_service_account.json> -n my-issuer-namespace

4. Create an issuer:

   1. Create a YAML file that defines the Issuer object:

      Example issuer.yaml file

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: <acme_dns01_clouddns_issuer> 1
        namespace: <issuer_namespace> 2
      spec:
        acme:
          preferredChain: ""
          privateKeySecretRef:
            name: <secret_private_key> 3
          server: https://acme-staging-v02.api.letsencrypt.org/directory 4
          solvers:
          - dns01:
              cloudDNS:
                project: <project_id> 5
                serviceAccountSecretRef:
                  name: clouddns-dns01-solver-svc-acct 6
                  key: service_account.json 7

      1

      Provide a name for the issuer.

      2

      Replace <issuer_namespace> with your issuer namespace.

      3

      Replace <secret_private_key> with the name of the secret to store the ACME account private key in.

      4

      Specify the URL to access the ACME server’s directory endpoint. This example uses the Let’s Encrypt staging environment.

      5

      Replace <project_id> with the name of the GCP project that contains the Cloud DNS zone.

      6

      Specify the name of the secret you created.

      7

      Specify the key in the secret you created that stores your GCP secret access key.

   2. Create the Issuer object by running the following command:

      $ oc create -f issuer.yaml

Procedure

1. Optional: Override the nameserver settings for the DNS-01 self check.

   This step is required only when the target public-hosted zone overlaps with the cluster’s default private-hosted zone.

   1. Edit the CertManager resource by running the following command:

      $ oc edit certmanager cluster

   2. Add a spec.controllerConfig section with the following override arguments:

      apiVersion: operator.openshift.io/v1alpha1
      kind: CertManager
      metadata:
        name: cluster
        ...
      spec:
        ...
        controllerConfig:                                1
          overrideArgs:
            - '--dns01-recursive-nameservers-only'       2
            - '--dns01-recursive-nameservers=1.1.1.1:53' 3

      1

      Add the spec.controllerConfig section.

      2

      Specify to only use recursive nameservers instead of checking the authoritative nameservers associated with that domain.

      3

      Provide a comma-separated list of <host>:<port> nameservers to query for the DNS-01 self check. You must use a 1.1.1.1:53 value to avoid the public and private zones overlapping.

   3. Save the file to apply the changes.

2. Optional: Create a namespace for the issuer:

   $ oc new-project <issuer_namespace>

3. Modify the CertManager resource to add the --issuer-ambient-credentials argument:

   $ oc patch certmanager/cluster \
     --type=merge \
     -p='{"spec":{"controllerConfig":{"overrideArgs":["--issuer-ambient-credentials"]}}}'

4. Create an issuer:

   1. Create a YAML file that defines the Issuer object:

      Example issuer.yaml file

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: <acme_dns01_clouddns_issuer> 1
        namespace: <issuer_namespace>
      spec:
        acme:
          preferredChain: ""
          privateKeySecretRef:
            name: <secret_private_key> 2
          server: https://acme-staging-v02.api.letsencrypt.org/directory 3
          solvers:
          - dns01:
              cloudDNS:
                project: <gcp_project_id> 4

      1

      Provide a name for the issuer.

      2

      Replace <secret_private_key> with the name of the secret to store the ACME account private key in.

      3

      Specify the URL to access the ACME server’s directory endpoint. This example uses the Let’s Encrypt staging environment.

      4

      Replace <gcp_project_id> with the name of the GCP project that contains the Cloud DNS zone.

   2. Create the Issuer object by running the following command:

      $ oc create -f issuer.yaml

Procedure

1. Optional: Override the nameserver settings for the DNS-01 self check.

   This step is required only when the target public-hosted zone overlaps with the cluster’s default private-hosted zone.

   1. Edit the CertManager resource by running the following command:

      $ oc edit certmanager cluster

   2. Add a spec.controllerConfig section with the following override arguments:

      apiVersion: operator.openshift.io/v1alpha1
      kind: CertManager
      metadata:
        name: cluster
        ...
      spec:
        ...
        controllerConfig:                                1
          overrideArgs:
            - '--dns01-recursive-nameservers-only'       2
            - '--dns01-recursive-nameservers=1.1.1.1:53' 3

      1

      Add the spec.controllerConfig section.

      2

      Specify to only use recursive nameservers instead of checking the authoritative nameservers associated with that domain.

      3

      Provide a comma-separated list of <host>:<port> nameservers to query for the DNS-01 self check. You must use a 1.1.1.1:53 value to avoid the public and private zones overlapping.

   3. Save the file to apply the changes.

2. Optional: Create a namespace for the issuer:

   $ oc new-project my-issuer-namespace

3. Create a secret to store your Azure credentials in by running the following command:

   $ oc create secret generic <secret_name> --from-literal=<azure_secret_access_key_name>=<azure_secret_access_key_value> \ 1 2 3
       -n my-issuer-namespace

   1

   Replace <secret_name> with your secret name.

   2

   Replace <azure_secret_access_key_name> with your Azure secret access key name.

   3

   Replace <azure_secret_access_key_value> with your Azure secret key.

4. Create an issuer:

   1. Create a YAML file that defines the Issuer object:

      Example issuer.yaml file

      apiVersion: cert-manager.io/v1
      kind: Issuer
      metadata:
        name: <acme-dns01-azuredns-issuer>   1
        namespace: <issuer_namespace>   2
      spec:
        acme:
          preferredChain: ""
          privateKeySecretRef:
            name: <secret_private_key> 3
          server: https://acme-staging-v02.api.letsencrypt.org/directory 4
          solvers:
          - dns01:
              azureDNS:
                clientID: <azure_client_id> 5
                clientSecretSecretRef:
                  name: <secret_name> 6
                  key: <azure_secret_access_key_name> 7
                subscriptionID: <azure_subscription_id> 8
                tenantID: <azure_tenant_id> 9
                resourceGroupName: <azure_dns_zone_resource_group> 10
                hostedZoneName: <azure_dns_zone> 11
                environment: AzurePublicCloud

      1

      Provide a name for the issuer.

      2

      Replace <issuer_namespace> with your issuer namespace.

      3

      Replace <secret_private_key> with the name of the secret to store the ACME account private key in.

      4

      Specify the URL to access the ACME server’s directory endpoint. This example uses the Let’s Encrypt staging environment.

      5

      Replace <azure_client_id> with your Azure client ID.

      6

      Replace <secret_name> with a name of the client secret.

      7

      Replace <azure_secret_access_key_name> with the client secret key name.

      8

      Replace <azure_subscription_id> with your Azure subscription ID.

      9

      Replace <azure_tenant_id> with your Azure tenant ID.

      10

      Replace <azure_dns_zone_resource_group> with the name of the Azure DNS zone resource group.

      11

      Replace <azure_dns_zone> with the name of Azure DNS zone.

   2. Create the Issuer object by running the following command:

      $ oc create -f issuer.yaml

Procedure

1. Create an issuer. For more information, see "Configuring an issuer" in the "Additional resources" section.

2. Create a certificate:

   1. Create a YAML file, for example, certificate.yaml, that defines the Certificate object:

      Example certificate.yaml file

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: <tls_cert> 1
        namespace: <issuer_namespace> 2
      spec:
        isCA: false
        commonName: '<common_name>' 3
        secretName: <secret_name> 4
        dnsNames:
        - "<domain_name>" 5
        issuerRef:
          name: <issuer_name> 6
          kind: Issuer

      1

      Provide a name for the certificate.

      2

      Specify the namespace of the issuer.

      3

      Specify the common name (CN).

      4

      Specify the name of the secret to create that contains the certificate.

      5

      Specify the domain name.

      6

      Specify the name of the issuer.

   2. Create the Certificate object by running the following command:

      $ oc create -f certificate.yaml

Procedure

1. Create an issuer. For more information, see "Configuring an issuer" in the "Additional resources" section.

2. Create a certificate:

   1. Create a YAML file, for example, certificate.yaml, that defines the Certificate object:

      Example certificate.yaml file

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: <tls_cert> 1
        namespace: openshift-config
      spec:
        isCA: false
        commonName: "api.<cluster_base_domain>" 2
        secretName: <secret_name> 3
        dnsNames:
        - "api.<cluster_base_domain>" 4
        issuerRef:
          name: <issuer_name> 5
          kind: Issuer

      1

      Provide a name for the certificate.

      2

      Specify the common name (CN).

      3

      Specify the name of the secret to create that contains the certificate.

      4

      Specify the DNS name of the API server.

      5

      Specify the name of the issuer.

   2. Create the Certificate object by running the following command:

      $ oc create -f certificate.yaml

3. Add the API server named certificate. For more information, see "Adding an API server named certificate" section in the "Additional resources" section.

Procedure

1. Create an issuer. For more information, see "Configuring an issuer" in the "Additional resources" section.

2. Create a certificate:

   1. Create a YAML file, for example, certificate.yaml, that defines the Certificate object:

      Example certificate.yaml file

      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: <tls_cert> 1
        namespace: openshift-ingress
      spec:
        isCA: false
        commonName: "apps.<cluster_base_domain>" 2
        secretName: <secret_name> 3
        dnsNames:
        - "apps.<cluster_base_domain>" 4
        - "*.apps.<cluster_base_domain>" 5
        issuerRef:
          name: <issuer_name> 6
          kind: Issuer

      1

      Provide a name for the certificate.

      2

      Specify the common name (CN).

      3

      Specify the name of the secret to create that contains the certificate.

      4 5

      Specify the DNS name of the ingress.

      6

      Specify the name of the issuer.

   2. Create the Certificate object by running the following command:

      $ oc create -f certificate.yaml

3. Replace the default ingress certificate. For more information, see "Replacing the default ingress certificate" section in the "Additional resources" section.

Procedure

1. Create a Route resource for your Service resource using edge TLS termination and a custom hostname by running the following command. The hostname will be used while creating a Certificate resource in the following steps.

   $ oc create route edge <route_name> \ 1
     --service=<service_name> \ 2
     --hostname=<hostname> \ 3
     --namespace=<namespace> 4

   1

   Specify your route’s name.

   2

   Specify the service you want to expose.

   3

   Specify the hostname of your route.

   4

   Specify the namespace where your route is located.

2. Create an Issuer to configure the HTTP-01 solver by running the following command. For other ACME issuer types, see "Configuring ACME an issuer".

   Example Issuer.yaml file

   $ oc create -f - << EOF
   apiVersion: cert-manager.io/v1
   kind: Issuer
   metadata:
     name: letsencrypt-acme
     namespace: <namespace> 1
   spec:
     acme:
       server: https://acme-v02.api.letsencrypt.org/directory
       privateKeySecretRef:
         name: letsencrypt-acme-account-key
       solvers:
         - http01:
             ingress:
               ingressClassName: openshift-default
   EOF

   1

   Specify the namespace where the Issuer is located. It should be the same as your route’s namespace.

3. Create a Certificate object for the route by running the following command. The secretName specifies the TLS secret that is going to be issued and managed by cert-manager and will also be referenced in your route in the following steps.

   $ oc create -f - << EOF
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: example-route-cert
     namespace: <namespace> 1
   spec:
     commonName: <hostname> 2
     dnsNames:
       - <hostname> 3
     usages:
       - server auth
     issuerRef:
       kind: Issuer
       name: letsencrypt-acme
     secretName: <secret_name> 4
   EOF

   1

   Specify the namespace where the Certificate resource is located. It should be the same as your route’s namespace.

   2

   Specify the certificate’s common name using the hostname of the route.

   3

   Add the hostname of your route to the certificate’s DNS names.

   4

   Specify the name of the secret that contains the certificate.

4. Create a Role to provide the router service account permissions to read the referenced secret by using the following command:

   $ oc create role secret-reader \
     --verb=get,list,watch \
     --resource=secrets \
     --resource-name=<secret_name> \ 1
     --namespace=<namespace> 2

   1

   Specify the name of the secret that you want to grant access to. It should be consistent with your secretName specified in the Certificate resource.

   2

   Specify the namespace where both your secret and route are located.

5. Create a RoleBinding resource to bind the router service account with the newly created Role resource by using the following command:

   $ oc create rolebinding secret-reader-binding \
     --role=secret-reader \
     --serviceaccount=openshift-ingress:router \
     --namespace=<namespace> 1

   1

   Specify the namespace where both your secret and route are located.

6. Update your route’s .spec.tls.externalCertificate field to reference the previously created secret and use the certificate issued by cert-manager by using the following command:

   $ oc patch route <route_name> \ 1
     -n <namespace> \ 2
     --type=merge \
     -p '{"spec":{"tls":{"externalCertificate":{"name":"<secret_name>"}}}}' 3

   1

   Specify the route name.

   2

   Specify the namespace where both your secret and route are located.

   3

   Specify the name of the secret that contains the certificate.

Procedure

1. Add the label to enable cluster monitoring by running the following command:

   $ oc label namespace cert-manager openshift.io/cluster-monitoring=true

2. Create a service monitor:

   1. Create a YAML file that defines the Role, RoleBinding, and ServiceMonitor objects:

      Example monitoring.yaml file

      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: prometheus-k8s
        namespace: cert-manager
      rules:
      - apiGroups:
        - ""
        resources:
        - services
        - endpoints
        - pods
        verbs:
        - get
        - list
        - watch
      ---
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: prometheus-k8s
        namespace: cert-manager
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: prometheus-k8s
      subjects:
      - kind: ServiceAccount
        name: prometheus-k8s
        namespace: openshift-monitoring
      ---
      apiVersion: monitoring.coreos.com/v1
      kind: ServiceMonitor
      metadata:
        labels:
          app: cert-manager
          app.kubernetes.io/component: controller
          app.kubernetes.io/instance: cert-manager
          app.kubernetes.io/name: cert-manager
        name: cert-manager
        namespace: cert-manager
      spec:
        endpoints:
        - interval: 30s
          port: tcp-prometheus-servicemonitor
          scheme: http
        selector:
          matchLabels:
            app.kubernetes.io/component: controller
            app.kubernetes.io/instance: cert-manager
            app.kubernetes.io/name: cert-manager

   2. Create the Role, RoleBinding, and ServiceMonitor objects by running the following command:

      $ oc create -f monitoring.yaml

Procedure

1. From the OpenShift Container Platform web console, navigate to Observe Metrics.

2. Add a query by using one of the following formats:

   • Specify the endpoints:

     {instance="<endpoint>"} 1

     1

     Replace <endpoint> with the value of the endpoint for the cert-manager service. You can find the endpoint value by running the following command: oc describe service cert-manager -n cert-manager.

   • Specify the tcp-prometheus-servicemonitor port:

     {endpoint="tcp-prometheus-servicemonitor"}

Procedure

1. Edit the CertManager resource by running the following command:

   $ oc edit certmanager.operator cluster

2. Set the log level value by editing the spec.logLevel section:

   apiVersion: operator.openshift.io/v1alpha1
   kind: CertManager
   ...
   spec:
     logLevel: <log_level> 1

   1

   The valid log level values for the CertManager resource are Normal, Debug, Trace, and TraceAll. To audit logs and perform common operations when there are no issues, set logLevel to Normal . To troubleshoot a minor issue by viewing verbose logs, set logLevel to Debug . To troubleshoot a major issue by viewing more verbose logs, you can set logLevel to Trace. To troubleshoot serious issues, set logLevel to TraceAll. The default logLevel is Normal.

   Note

   TraceAll generates huge amount of logs. After setting logLevel to TraceAll, you might experience performance issues.

3. Save your changes and quit the text editor to apply your changes.

   After applying the changes, the verbosity level for the cert-manager components controller, CA injector, and webhook is updated.

Verification

1. The cert-manager Operator pod is redeployed. Verify that the log level of the cert-manager Operator for Red Hat OpenShift is updated by running the following command:

   $ oc set env deploy/cert-manager-operator-controller-manager -n cert-manager-operator --list | grep -e OPERATOR_LOG_LEVEL -e container

   Example output

   # deployments/cert-manager-operator-controller-manager, container kube-rbac-proxy
   OPERATOR_LOG_LEVEL=9
   # deployments/cert-manager-operator-controller-manager, container cert-manager-operator
   OPERATOR_LOG_LEVEL=9

2. Verify that the log level of the cert-manager Operator for Red Hat OpenShift is updated by running the oc logs command:

   $ oc logs deploy/cert-manager-operator-controller-manager -n cert-manager-operator

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Uninstall the cert-manager Operator for Red Hat OpenShift Operator.

   1. Navigate to Operators Installed Operators.
   2. Click the Options menu next to the cert-manager Operator for Red Hat OpenShift entry and click Uninstall Operator.
   3. In the confirmation dialog, click Uninstall.

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Remove the deployments of the cert-manager components, such as cert-manager, cainjector, and webhook, present in the cert-manager namespace.

   1. Click the Project drop-down menu to see a list of all available projects, and select the cert-manager project.
   2. Navigate to Workloads Deployments.
   3. Select the deployment that you want to delete.
   4. Click the Actions drop-down menu, and select Delete Deployment to see a confirmation dialog box.
   5. Click Delete to delete the deployment.

   6. Alternatively, delete deployments of the cert-manager components such as cert-manager, cainjector and webhook present in the cert-manager namespace by using the command-line interface (CLI).

      $ oc delete deployment -n cert-manager -l app.kubernetes.io/instance=cert-manager

3. Optional: Remove the custom resource definitions (CRDs) that were installed by the cert-manager Operator for Red Hat OpenShift:

   1. Navigate to Administration CustomResourceDefinitions.
   2. Enter certmanager in the Name field to filter the CRDs.

   3. Click the Options menu next to each of the following CRDs, and select Delete Custom Resource Definition:

      • Certificate
      • CertificateRequest
      • CertManager (operator.openshift.io)
      • Challenge
      • ClusterIssuer
      • Issuer
      • Order

4. Optional: Remove the cert-manager-operator namespace.

   1. Navigate to Administration Namespaces.
   2. Click the Options menu next to the cert-manager-operator and select Delete Namespace.
   3. In the confirmation dialog, enter cert-manager-operator in the field and click Delete.