Dieser Inhalt ist in der von Ihnen ausgewählten Sprache nicht verfügbar.
Chapter 14. Tutorial: Dynamically issuing certificates using the cert-manager Operator on ROSA
While wildcard certificates provide simplicity by securing all first-level subdomains of a given domain with a single certificate, other use cases can require the use of individual certificates per domain.
Learn how to use the cert-manager Operator for Red Hat OpenShift and Let’s Encrypt to dynamically issue certificates for routes created using a custom domain.
14.1. Prerequisites
- A ROSA cluster (HCP or Classic)
-
A user account with
cluster-admin
privileges -
The OpenShift CLI (
oc
) -
The Amazon Web Services (AWS) CLI (
aws
) -
A unique domain, such as
*.apps.example.com
- An Amazon Route 53 public hosted zone for the above domain
14.2. Setting up your environment
Configure the following environment variables:
$ export DOMAIN=apps.example.com 1 $ export EMAIL=email@example.com 2 $ export AWS_PAGER="" $ export CLUSTER=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}" | sed 's/-[a-z0-9]\{5\}$//') $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed 's|^https://||') $ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}") $ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text) $ export SCRATCH="/tmp/${CLUSTER}/dynamic-certs" $ mkdir -p ${SCRATCH}
Ensure all fields output correctly before moving to the next section:
$ echo "Cluster: ${CLUSTER}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"
NoteThe "Cluster" output from the previous command may be the name of your cluster, the internal ID of your cluster, or the cluster’s domain prefix. If you prefer to use another identifier, you can manually set this value by running the following command:
$ export CLUSTER=my-custom-value
14.3. Preparing your AWS account
When cert-manager requests a certificate from Let’s Encrypt (or another ACME certificate issuer), Let’s Encrypt servers validate that you control the domain name in that certificate using challenges. For this tutorial, you are using a DNS-01 challenge that proves that you control the DNS for your domain name by putting a specific value in a TXT record under that domain name. This is all done automatically by cert-manager. To allow cert-manager permission to modify the Amazon Route 53 public hosted zone for your domain, you need to create an Identity Access Management (IAM) role with specific policy permissions and a trust relationship to allow access to the pod.
The public hosted zone that is used in this tutorial is in the same AWS account as the ROSA cluster. If your public hosted zone is in a different account, a few additional steps for Cross Account Access are required.
Retrieve the Amazon Route 53 public hosted zone ID:
NoteThis command looks for a public hosted zone that matches the custom domain you specified earlier as the
DOMAIN
environment variable. You can manually specify the Amazon Route 53 public hosted zone by runningexport ZONE_ID=<zone_ID>
, replacing<zone_ID>
with your specific Amazon Route 53 public hosted zone ID.$ export ZONE_ID=$(aws route53 list-hosted-zones-by-name --output json \ --dns-name "${DOMAIN}." --query 'HostedZones[0]'.Id --out text | sed 's/\/hostedzone\///')
Create an AWS IAM policy document for the cert-manager Operator that provides the ability to update only the specified public hosted zone:
$ cat <<EOF > "${SCRATCH}/cert-manager-policy.json" { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "route53:GetChange", "Resource": "arn:aws:route53:::change/*" }, { "Effect": "Allow", "Action": [ "route53:ChangeResourceRecordSets", "route53:ListResourceRecordSets" ], "Resource": "arn:aws:route53:::hostedzone/${ZONE_ID}" }, { "Effect": "Allow", "Action": "route53:ListHostedZonesByName", "Resource": "*" } ] } EOF
Create the IAM policy using the file you created in the previous step:
$ POLICY_ARN=$(aws iam create-policy --policy-name "${CLUSTER}-cert-manager-policy" \ --policy-document file://${SCRATCH}/cert-manager-policy.json \ --query 'Policy.Arn' --output text)
Create an AWS IAM trust policy for the cert-manager Operator:
$ cat <<EOF > "${SCRATCH}/trust-policy.json" { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Condition": { "StringEquals" : { "${OIDC_ENDPOINT}:sub": "system:serviceaccount:cert-manager:cert-manager" } }, "Principal": { "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}" }, "Action": "sts:AssumeRoleWithWebIdentity" } ] } EOF
Create an IAM role for the cert-manager Operator using the trust policy you created in the previous step:
$ ROLE_ARN=$(aws iam create-role --role-name "${CLUSTER}-cert-manager-operator" \ --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \ --query Role.Arn --output text)
Attach the permissions policy to the role:
$ aws iam attach-role-policy --role-name "${CLUSTER}-cert-manager-operator" \ --policy-arn ${POLICY_ARN}
14.4. Installing the cert-manager Operator
Create a project to install the cert-manager Operator into:
$ oc new-project cert-manager-operator
ImportantDo not attempt to use more than one cert-manager Operator in your cluster. If you have a community cert-manager Operator installed in your cluster, you must uninstall it before installing the cert-manager Operator for Red Hat OpenShift.
Install the cert-manager Operator for Red Hat OpenShift:
$ cat << EOF | oc apply -f - apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-cert-manager-operator-group namespace: cert-manager-operator spec: targetNamespaces: - cert-manager-operator --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: openshift-cert-manager-operator namespace: cert-manager-operator spec: channel: stable-v1 installPlanApproval: Automatic name: openshift-cert-manager-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
NoteIt takes a few minutes for this Operator to install and complete its set up.
Verify that the cert-manager Operator is running:
$ oc -n cert-manager-operator get pods
Example output
NAME READY STATUS RESTARTS AGE cert-manager-operator-controller-manager-84b8799db5-gv8mx 2/2 Running 0 12s
Annotate the service account used by the cert-manager pods with the AWS IAM role you created earlier:
$ oc -n cert-manager annotate serviceaccount cert-manager eks.amazonaws.com/role-arn=${ROLE_ARN}
Restart the existing cert-manager controller pod by running the following command:
$ oc -n cert-manager delete pods -l app.kubernetes.io/name=cert-manager
Patch the Operator’s configuration to use external nameservers to prevent DNS-01 challenge resolution issues:
$ oc patch certmanager.operator.openshift.io/cluster --type merge \ -p '{"spec":{"controllerConfig":{"overrideArgs":["--dns01-recursive-nameservers-only","--dns01-recursive-nameservers=1.1.1.1:53"]}}}'
Create a
ClusterIssuer
resource to use Let’s Encrypt by running the following command:$ cat << EOF | oc apply -f - apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-production spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: ${EMAIL} # This key doesn't exist, cert-manager creates it privateKeySecretRef: name: prod-letsencrypt-issuer-account-key solvers: - dns01: route53: hostedZoneID: ${ZONE_ID} region: ${REGION} secretAccessKeySecretRef: name: '' EOF
Verify the
ClusterIssuer
resource is ready:$ oc get clusterissuer.cert-manager.io/letsencrypt-production
Example output
NAME READY AGE letsencrypt-production True 47s
14.5. Creating a custom domain Ingress Controller
Create and configure a certificate resource to provision a certificate for the custom domain Ingress Controller:
NoteThe following example uses a single domain certificate. SAN and wildcard certificates are also supported.
$ cat << EOF | oc apply -f - apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: custom-domain-ingress-cert namespace: openshift-ingress spec: secretName: custom-domain-ingress-cert-tls issuerRef: name: letsencrypt-production kind: ClusterIssuer commonName: "${DOMAIN}" dnsNames: - "${DOMAIN}" EOF
Verify the certificate has been issued:
NoteIt takes a few minutes for this certificate to be issued by Let’s Encrypt. If it takes longer than 5 minutes, run
oc -n openshift-ingress describe certificate.cert-manager.io/custom-domain-ingress-cert
to see any issues reported by cert-manager.$ oc -n openshift-ingress get certificate.cert-manager.io/custom-domain-ingress-cert
Example output
NAME READY SECRET AGE custom-domain-ingress-cert True custom-domain-ingress-cert-tls 9m53s
Create a new
IngressController
resource:$ cat << EOF | oc apply -f - apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: custom-domain-ingress namespace: openshift-ingress-operator spec: domain: ${DOMAIN} defaultCertificate: name: custom-domain-ingress-cert-tls endpointPublishingStrategy: loadBalancer: dnsManagementPolicy: Unmanaged providerParameters: aws: type: NLB type: AWS scope: External type: LoadBalancerService EOF
WarningThis
IngressController
example will create an internet accessible Network Load Balancer (NLB) in your AWS account. To provision an internal NLB instead, set the.spec.endpointPublishingStrategy.loadBalancer.scope
parameter toInternal
before creating theIngressController
resource.Verify that your custom domain IngressController has successfully created an external load balancer:
$ oc -n openshift-ingress get service/router-custom-domain-ingress
Example output
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE router-custom-domain-ingress LoadBalancer 172.30.174.34 a309962c3bd6e42c08cadb9202eca683-1f5bbb64a1f1ec65.elb.us-east-1.amazonaws.com 80:31342/TCP,443:31821/TCP 7m28s
Prepare a document with the necessary DNS changes to enable DNS resolution for your custom domain Ingress Controller:
$ INGRESS=$(oc -n openshift-ingress get service/router-custom-domain-ingress -ojsonpath="{.status.loadBalancer.ingress[0].hostname}") $ cat << EOF > "${SCRATCH}/create-cname.json" { "Comment":"Add CNAME to custom domain endpoint", "Changes":[{ "Action":"CREATE", "ResourceRecordSet":{ "Name": "*.${DOMAIN}", "Type":"CNAME", "TTL":30, "ResourceRecords":[{ "Value": "${INGRESS}" }] } }] } EOF
Submit your changes to Amazon Route 53 for propagation:
$ aws route53 change-resource-record-sets \ --hosted-zone-id ${ZONE_ID} \ --change-batch file://${SCRATCH}/create-cname.json
NoteWhile the wildcard CNAME record avoids the need to create a new record for every new application you deploy using the custom domain Ingress Controller, the certificate that each of these applications use is not a wildcard certificate.
14.6. Configuring dynamic certificates for custom domain routes
Now you can expose cluster applications on any first-level subdomains of the specified domain, but the connection will not be secured with a TLS certificate that matches the domain of the application. To ensure these cluster applications have valid certificates for each domain name, configure cert-manager to dynamically issue a certificate to every new route created under this domain.
Create the necessary OpenShift resources cert-manager requires to manage certificates for OpenShift routes.
This step creates a new deployment (and therefore a pod) that specifically monitors annotated routes in the cluster. If the
issuer-kind
andissuer-name
annotations are found in a new route, it requests the Issuer (ClusterIssuer in this case) for a new certificate that is unique to this route and which will honor the hostname that was specified while creating the route.NoteIf the cluster does not have access to GitHub, you can save the raw contents locally and run
oc apply -f localfilename.yaml -n cert-manager
.$ oc -n cert-manager apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml
The following additional OpenShift resources are also created in this step:
-
ClusterRole
- grants permissions to watch and update the routes across the cluster -
ServiceAccount
- uses permissions to run the newly created pod -
ClusterRoleBinding
- binds these two resources
-
Ensure that the new
cert-manager-openshift-routes
pod is running successfully:$ oc -n cert-manager get pods
Example result
NAME READY STATUS RESTARTS AGE cert-manager-866d8f788c-9kspc 1/1 Running 0 4h21m cert-manager-cainjector-6885c585bd-znws8 1/1 Running 0 4h41m cert-manager-openshift-routes-75b6bb44cd-f8kd5 1/1 Running 0 6s cert-manager-webhook-8498785dd9-bvfdf 1/1 Running 0 4h41m
14.7. Deploying a sample application
Now that dynamic certificates are configured, you can deploy a sample application to confirm that certificates are provisioned and trusted when you expose a new route.
Create a new project for your sample application:
$ oc new-project hello-world
Deploy a hello world application:
$ oc -n hello-world new-app --image=docker.io/openshift/hello-openshift
Create a route to expose the application from outside the cluster:
$ oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls --hostname hello.${DOMAIN}
Verify the certificate for the route is untrusted:
$ curl -I https://hello.${DOMAIN}
Example output
curl: (60) SSL: no alternative certificate subject name matches target host name 'hello.example.com' More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.
Annotate the route to trigger cert-manager to provision a certificate for the custom domain:
$ oc -n hello-world annotate route hello-openshift-tls cert-manager.io/issuer-kind=ClusterIssuer cert-manager.io/issuer-name=letsencrypt-production
NoteIt takes 2-3 minutes for the certificate to be created. The renewal of the certificate will automatically be managed by the cert-manager Operator as it approaches expiration.
Verify the certificate for the route is now trusted:
$ curl -I https://hello.${DOMAIN}
Example output
HTTP/2 200 date: Thu, 05 Oct 2023 23:45:33 GMT content-length: 17 content-type: text/plain; charset=utf-8 set-cookie: 52e4465485b6fb4f8a1b1bed128d0f3b=68676068bb32d24f0f558f094ed8e4d7; path=/; HttpOnly; Secure; SameSite=None cache-control: private
14.8. Troubleshooting dynamic certificate provisioning
The validation process usually takes 2-3 minutes to complete while creating certificates.
If annotating your route does not trigger certificate creation during the certificate create step, run oc describe
against each of the certificate
,certificaterequest
,order
, and challenge
resources to view the events or reasons that can help identify the cause of the issue.
$ oc get certificate,certificaterequest,order,challenge
For troubleshooting, you can refer to this helpful guide in debugging certificates.
You can also use the cmctl CLI tool for various certificate management activities, such as checking the status of certificates and testing renewals.