Tutorials

1. Activate the ROSA with HCP product at the AWS console page by clicking the Get started button:

   Figure 2.1. Get started

   

   If you have activated ROSA before but did not complete the process, you can click the button and complete the account linking as described in the following steps.

2. Confirm that you want your contact information to be shared with Red Hat and enable the service:

   Figure 2.2. Enable ROSA

   
   • You will not be charged by enabling the service in this step. The connection is made for billing and metering that will take place only after you deploy your first cluster. This could take a few minutes.

3. After the process is completed, you will see a confirmation:

   Figure 2.3. ROSA enablement confirmation

   

4. Other sections on this verification page show the status of additional prerequisites. In case any of these prerequisites are not met, a corresponding message is shown. Here is an example of insufficient quotas in the selected region:

   Figure 2.4. Service quotas

   
   1. Click the Increase service quotas button or use the Learn more link to get more information about the about how to manage service quotas. In the case of insufficient quotas, note that quotas are region-specific. You can use the region switcher in the upper right corner of the web console to re-run the quota check for any region you are interested in and then submit service quota increase requests as needed.

5. If all the prerequisites are met, the page will look like this:

   Figure 2.5. Verify ROSA prerequisites

   

   The ELB service-linked role is created for you automatically. You can click any of the small Info blue links to get contextual help and resources.

1. Click the orange Continue to Red Hat button to proceed with account linking:

   Figure 2.6. Continue to Red Hat

   

2. If you are not already logged in to your Red Hat account in your current browser’s session, you will be asked to log in to your account:

   Note

   Your AWS account must be linked to a single Red Hat organization.

   Figure 2.7. Log in to your Red Hat account

   
   • You can also register for a new Red Hat account or reset your password on this page.
   • Make sure to log in to the Red Hat account that you plan to associate with the AWS account where you have activated ROSA with HCP in previous steps.
   • The AWS account used for service billing can only be associated with a single Red Hat account. Typically an AWS payer account is the one that is used to subscribe to ROSA and used for account linking and billing.
   • All team members belonging to the same Red Hat organization can use the linked AWS account for service billing while creating ROSA with HCP clusters.

3. Complete the Red Hat account linking after reviewing the terms and conditions:

   Note

   This step is available only if the AWS account was not linked to any Red Hat account before.

   This step is skipped if the AWS account is already linked to the user’s logged in Red Hat account.

   If the AWS account is linked to a different Red Hat account, an error will be displayed. See Correcting Billing Account Information for HCP clusters for troubleshooting.

   Figure 2.8. Complete your account connection

   

   Both the Red Hat and AWS account numbers are shown on this screen.

4. Click the Connect accounts button if you agree with the service terms.

   If this is the first time you are using the Red Hat Hybrid Cloud Console, you will be asked to agree with the general managed services terms and conditions before being able to create the first ROSA cluster:

   Figure 2.9. Terms and conditions

   

   Additional terms that need to be reviewed and accepted are shown after clicking the View Terms and Conditions button:

   Figure 2.10. Red Hat terms and conditions

   

   Submit your agreement once you have reviewed any additional terms when prompted at this time.

5. The Hybrid Cloud Console provides a confirmation that AWS account setup was completed and lists the prerequisites for cluster deployment:

   Figure 2.11. Complete ROSA prerequisites

   

   The last section of this page shows cluster deployment options, either using the rosa CLI or through the web console:

   Figure 2.12. Deploy the cluster and set up access

   

1. Initiate the cluster deployment using the rosa create cluster command. You can click the copy button on the Set up Red Hat OpenShift Service on AWS (ROSA) console page and paste the command in your terminal. This launches the cluster creation process in interactive mode:

   Figure 2.13. Deploy the cluster and set up access

   
2. To use a custom AWS profile, one of the non-default profiles specified in your ~/.aws/credentials, you can add the –profile <profile_name> selector to the rosa create cluster command so that the command looks like rosa create cluster –profile stage. If no AWS CLI profile is specified using this option, the default AWS CLI profile will determine the AWS infrastructure profile into which the cluster is deployed. The billing AWS profile is selected in one of the following steps.

3. When deploying a ROSA with HCP cluster, the billing AWS account needs to be specified:

   Figure 2.14. Specify the Billing Account

   
   • Only AWS accounts that are linked to the user’s logged in Red Hat account are shown.
   • The specified AWS account is charged for using the ROSA service.

   • An indicator shows if the ROSA contract is enabled or not enabled for a given AWS billing account.

     • If you select an AWS billing account that shows the Contract enabled label, on-demand consumption rates are charged only after the capacity of your pre-paid contract is consumed.
     • AWS accounts without the Contract enabled label are charged the applicable on-demand consumption rates.

1. A cluster can be created using the web console by selecting the second option in the bottom section of the introductory Set up ROSA page:

   Figure 2.15. Deploy with web interface

   
   Note

   Complete the prerequisites before starting the web console deployment process.

   The rosa CLI is required for certain tasks, such as creating the account roles. If you are deploying ROSA for the first time, follow this the CLI steps until running the rosa whoami command, before starting the web console deployment steps.

2. The first step when creating a ROSA cluster using the web console is the control plane selection. Make sure the Hosted option is selected before clicking the Next button:

   Figure 2.16. Select hosted option

   

3. The next step Accounts and roles allows you specifying the infrastructure AWS account, into which the ROSA cluster is deployed and where the resources are consumed and managed:

   Figure 2.17. AWS infrastructure account

   
   • Click the How to associate a new AWS account, if you don not see the account into which you want to deploy the ROSA cluster for detailed information on how to create or link account roles for this association.
   • The rosa CLI is used for this.
   • If you are using multiple AWS accounts and have their profiles configured for the AWS CLI, you can use the --profile selector to specify the AWS profile when working with the rosa CLI commands.

4. The billing AWS account is selected in the immediately following section:

   Figure 2.18. AWS billing account

   
   • Only AWS accounts that are linked to the user’s logged in Red Hat account are shown.
   • The specified AWS account is charged for using the ROSA service.

   • An indicator shows if the ROSA contract is enabled or not enabled for a given AWS billing account.

     • If you select an AWS billing account that shows the Contract enabled label, on-demand consumption rates are charged only after the capacity of your pre-paid contract is consumed.
     • AWS accounts without the Contract enabled label are charged the applicable on-demand consumption rates.

1. When you get a private offer for ROSA with HCP, you are provided with a unique URL that is accessible only by a specific AWS account ID that was specified by the seller.

   Note

   Verify that you are logged in using the AWS account that was specified as the buyer. Attempting to access the offer using another AWS account produces a "page not found" error message as shown in Figure 11 in the troubleshooting section below.

   1. You can see the offer selection drop down menu with a regular private offer pre-selected in Figure 1. This type of offer can be accepted only if the ROSA with HCP was not activated before using the public offer or another private offer.

      Figure 3.1. Regular private offer

      

   2. You can see a private offer that was created for an AWS account that previously activated ROSA with HCP using the public offer, showing the product name and the selected private offer labeled as "Upgrade", that replaces the currently running contract for ROSA with HCP in Figure 2.

      Figure 3.2. Private offer selection selection screen

      

   3. The drop down menu allows selecting between multiple offers, if available. The previously activated public offer is shown together with the newly provided agreement based offer that is labeled as "Upgrade" in Figure 3.

      Figure 3.3. Private offer selection dropdown

      

2. Verify that your offer configuration is selected. Figure 4 shows the bottom part of the offer page with the offer details.

   Note

   The contract end date, the number of units included with the offer, and the payment schedule. In this example, 1 cluster and up to 3 nodes utilizing 4 vCPUs are included.

   Figure 3.4. Private offer details

   

3. Optional: you can add your own purchase order (PO) number to the subscription that is being purchased, so it is included on your subsequent AWS invoices. Also, check the "Additional usage fees" that are charged for any usage above the scope of the "New offer configuration details".

   Note

   Private offers have several available configurations.

   • It is possible that the private offer you are accepting is set up with a fixed future start date.
   • If you do not have another active ROSA with HCP subscription at the time of accepting the private offer, a public offer or an older private offer entitlement, accept the private offer itself and continue with the account linking and cluster deployment steps after the specified service start date.

   You must have an active ROSA with HCP entitlement to complete these steps. Service start dates are always reported in the UTC time zone

4. Create or upgrade your contract.

   1. For private offers accepted by an AWS account that does not have ROSA with HCP activated yet and is creating the first contract for this service, click the Create contract button.

      Figure 3.5. Create contract button

      

   2. For agreement-based offers, click the Upgrade current contract button shown in Figures 4 and 6.

      Figure 3.6. Upgrade contract button

      

5. Click Confirm.

   Figure 3.7. Private offer acceptance confirmation window

   

6. If the accepted private offer service start date is set to be immediately following the offer acceptance, click the Set up your account button in the confirmation modal window.

   Figure 3.8. Subscription confirmation

   

7. If the accepted private offer has a future start date specified, return to the private offer page after the service start date, and click the Setup your account button to proceed with the Red Hat and AWS account linking.

   Note

   With no agreement active, the account linking described below is not triggered, the "Account setup" process can be done only after the "Service start date".

   These are always in UTC time zone.

1. Clicking the Set up your account button in the previous step takes you to the AWS and Red Hat account linking step. At this time, you are already logged in with the AWS account that accepted the offer. If you are not logged in with a Red Hat account, you will be prompted to do so.

   ROSA with HCP entitlement is shared with other team members through your Red Hat organization account. All existing users in the same Red Hat organization are able to select the billing AWS account that accepted the private offer by following the above described steps. You can manage users in your Red Hat organization, when logged in as the Red Hat organization administrator, and invite or create new users.

   Note

   ROSA with HCP private offer cannot be shared with AWS linked accounts through the AWS License Manager.

2. Add any users that you want to deploy ROSA clusters. Check this user management FAQ for more details about Red Hat account user management tasks.
3. Verify that the already logged in Red Hat account includes all users that are meant to be ROSA cluster deployers benefiting from the accepted private offer.

4. Verify that the Red Hat account number and the AWS account ID are the desired accounts that are to be linked. This linking is unique and a Red Hat account can be connected only with a single AWS (billing) account.

   Figure 3.9. AWS and Red Hat accounts connection

   

5. If you want to link the AWS account with another Red Hat account than is shown on this page in Figure 9, log out from the Red Hat Hybrid Cloud Console before connecting the accounts and repeat the step of setting the account by returning to the private offer URL that is already accepted.

   An AWS account can be connected with a single Red Hat account only. Once Red Hat and AWS accounts are connected, this cannot be changed by the user. If a change is needed, the user must create a support ticket.

6. Agree to the terms and conditions and then click Connect accounts.

1. To use the script, run the following commands in a bash terminal (the -p option defines a prefix for the roles):

   $ mkdir scratch
   $ cd scratch
   $ cat << 'EOF' > verify-permissions.sh
   #!/bin/bash
   while getopts 'p:' OPTION; do
     case "$OPTION" in
       p)
         PREFIX="$OPTARG"
         ;;
       ?)
         echo "script usage: $(basename \$0) [-p PREFIX]" >&2
         exit 1
         ;;
     esac
   done
   shift "$(($OPTIND -1))"
   rosa create account-roles --mode manual --prefix $PREFIX
   INSTALLER_POLICY=$(cat sts_installer_permission_policy.json | jq )
   CONTROL_PLANE_POLICY=$(cat sts_instance_controlplane_permission_policy.json | jq)
   WORKER_POLICY=$(cat sts_instance_worker_permission_policy.json | jq)
   SUPPORT_POLICY=$(cat sts_support_permission_policy.json | jq)
   simulatePolicy () {
       outputFile="${2}.results"
       echo $2
       aws iam simulate-custom-policy --policy-input-list "$1" --action-names $(jq '.Statement | map(select(.Effect == "Allow"))[].Action | if type == "string" then . else .[] end' "$2" -r) --output text > $outputFile
   }
   simulatePolicy "$INSTALLER_POLICY" "sts_installer_permission_policy.json"
   simulatePolicy "$CONTROL_PLANE_POLICY" "sts_instance_controlplane_permission_policy.json"
   simulatePolicy "$WORKER_POLICY" "sts_instance_worker_permission_policy.json"
   simulatePolicy "$SUPPORT_POLICY" "sts_support_permission_policy.json"
   EOF
   $ chmod +x verify-permissions.sh
   $ ./verify-permissions.sh -p SimPolTest

2. After the script completes, review each results file to ensure that none of the required API calls are blocked:

   $ for file in $(ls *.results); do echo $file; cat $file; done

   The output will look similar to the following:

   sts_installer_permission_policy.json.results
   EVALUATIONRESULTS       autoscaling:DescribeAutoScalingGroups   allowed *
   MATCHEDSTATEMENTS       PolicyInputList.1       IAM Policy
   ENDPOSITION     6       195
   STARTPOSITION   17      3
   EVALUATIONRESULTS       ec2:AllocateAddress     allowed *
   MATCHEDSTATEMENTS       PolicyInputList.1       IAM Policy
   ENDPOSITION     6       195
   STARTPOSITION   17      3
   EVALUATIONRESULTS       ec2:AssociateAddress    allowed *
   MATCHEDSTATEMENTS       PolicyInputList.1       IAM Policy
   ...

   Note

   If any actions are blocked, review the error provided by AWS and consult with your Administrator to determine if SCPs are blocking the required API calls.

1. Configure the following environment variables:

   $ export VPC_ID=<vpc_ID> 1
   $ export REGION=<region> 2
   $ export VPC_CIDR=<vpc_CIDR> 3

   1

   Replace <vpc_ID> with the ID of the VPC you want to install your cluster into.

   2

   Replace <region> with the AWS region you want to install your cluster into.

   3

   Replace <vpc_CIDR> with the CIDR range of your VPC.

2. Ensure all fields output correctly before moving to the next section:

   $ echo "VPC ID: ${VPC_ID}, VPC CIDR Range: ${VPC_CIDR}, Region: ${REGION}"

1. Create a security group and allow access to ports 53/tcp and 53/udp from the VPC:

   $ SG_ID=$(aws ec2 create-security-group --group-name rosa-inbound-resolver --description "Security group for ROSA inbound resolver" --vpc-id ${VPC_ID} --region ${REGION} --output text)
   $ aws ec2 authorize-security-group-ingress --group-id ${SG_ID} --protocol tcp --port 53 --cidr ${VPC_CIDR} --region ${REGION}
   $ aws ec2 authorize-security-group-ingress --group-id ${SG_ID} --protocol udp --port 53 --cidr ${VPC_CIDR} --region ${REGION}

2. Create an Amazon Route 53 Inbound Resolver in your VPC:

   $ RESOLVER_ID=$(aws route53resolver create-resolver-endpoint \
     --name rosa-inbound-resolver \
     --creator-request-id rosa-$(date '+%Y-%m-%d') \
     --security-group-ids ${SG_ID} \
     --direction INBOUND \
     --ip-addresses $(aws ec2 describe-subnets --filter Name=vpc-id,Values=${VPC_ID} --region ${REGION} | jq -jr '.Subnets | map("SubnetId=\(.SubnetId) ") | .[]') \
     --region ${REGION} \
     --output text \
     --query 'ResolverEndpoint.Id')

   Note

   The above command attaches Amazon Route 53 Inbound Resolver endpoints to all subnets in the provided VPC using dynamically allocated IP addresses. If you prefer to manually specify the subnets and/or IP addresses, run the following command instead:

   $ RESOLVER_ID=$(aws route53resolver create-resolver-endpoint \
     --name rosa-inbound-resolver \
     --creator-request-id rosa-$(date '+%Y-%m-%d') \
     --security-group-ids ${SG_ID} \
     --direction INBOUND \
     --ip-addresses SubnetId=<subnet_ID>,Ip=<endpoint_IP> SubnetId=<subnet_ID>,Ip=<endpoint_IP> \1
     --region ${REGION} \
     --output text \
     --query 'ResolverEndpoint.Id')

   1

   Replace <subnet_ID> with the subnet IDs and <endpoint_IP> with the static IP addresses you want inbound resolver endpoints added to.

3. Get the IP addresses of your inbound resolver endpoints to configure in your DNS server configuration:

   $ aws route53resolver list-resolver-endpoint-ip-addresses \
     --resolver-endpoint-id ${RESOLVER_ID} \
     --region=${REGION} \
     --query 'IpAddresses[*].Ip'

   Example output

   [
       "10.0.45.253",
       "10.0.23.131",
       "10.0.148.159"
   ]

Procedure

1. Create a new TLS secret from a private key and a public certificate, where fullchain.pem is your full wildcard certificate chain (including any intermediaries) and privkey.pem is your wildcard certificate’s private key.

   Example

   $ oc -n openshift-ingress create secret tls waf-tls --cert=fullchain.pem --key=privkey.pem

2. Create a new IngressController resource:

   Example waf-ingress-controller.yaml

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: cloudfront-waf
     namespace: openshift-ingress-operator
   spec:
     domain: apps.example.com 1
     defaultCertificate:
       name: waf-tls
     endpointPublishingStrategy:
       loadBalancer:
         dnsManagementPolicy: Unmanaged
         providerParameters:
           aws:
             type: NLB
           type: AWS
         scope: External
       type: LoadBalancerService
     routeSelector: 2
       matchLabels:
        route: waf

   1

   Replace with the custom domain you want to use for the IngressController.

   2

   Filters the set of routes serviced by the Ingress Controller. In this tutorial, we will use the waf route selector, but if no value was to be provided, no filtering would occur.

3. Apply the IngressController:

   Example

   $ oc apply -f waf-ingress-controller.yaml

4. Verify that your IngressController has successfully created an external load balancer:

   $ oc -n openshift-ingress get service/router-cloudfront-waf

   Example output

   NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)                      AGE
   router-cloudfront-waf   LoadBalancer   172.30.16.141   a68a838a7f26440bf8647809b61c4bc8-4225395f488830bd.elb.us-east-1.amazonaws.com   80:30606/TCP,443:31065/TCP   2m19s

1. Retrieve the newly created custom ingress controller’s NLB hostname:

   $ NLB=$(oc -n openshift-ingress get service router-cloudfront-waf \
     -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

2. Import your certificate into Amazon Certificate Manager, where cert.pem is your wildcard certificate, fullchain.pem is your wildcard certificate’s chain and privkey.pem is your wildcard certificate’s private key.

   Note

   Regardless of what region your cluster is deployed, you must import this certificate to us-east-1 as Amazon CloudFront is a global AWS service.

   Example

   $ aws acm import-certificate --certificate file://cert.pem \
     --certificate-chain file://fullchain.pem \
     --private-key file://privkey.pem \
     --region us-east-1

3. Log into the AWS console to create a CloudFront distribution.

4. Configure the CloudFront distribution by using the following information:

   Note

   If an option is not specified in the table below, leave them the default (which may be blank).

   Option	Value
   Origin domain	Output from the previous command [1]
   Name	rosa-waf-ingress [2]
   Viewer protocol policy	Redirect HTTP to HTTPS
   Allowed HTTP methods	GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE
   Cache policy	CachingDisabled
   Origin request policy	AllViewer
   Web Application Firewall (WAF)	Enable security protections
   Use existing WAF configuration	true
   Choose a web ACL	cloudfront-waf
   Alternate domain name (CNAME)	*.apps.example.com [3]
   Custom SSL certificate	Select the certificate you imported from the step above [4]

   1. Run echo ${NLB} to get the origin domain.
   2. If you have multiple clusters, ensure the origin name is unique.
   3. This should match the wildcard domain you used to create the custom ingress controller.
   4. This should match the alternate domain name entered above.

5. Retrieve the Amazon CloudFront Distribution endpoint:

   $ aws cloudfront list-distributions --query "DistributionList.Items[?Origins.Items[?DomainName=='${NLB}']].DomainName" --output text

6. Update the DNS of your custom wildcard domain with a CNAME to the Amazon CloudFront Distribution endpoint from the step above.

   Example

   *.apps.example.com CNAME d1b2c3d4e5f6g7.cloudfront.net

1. Create a new project for your sample application by running the following command:

   $ oc new-project hello-world

2. Deploy a hello world application:

   $ oc -n hello-world new-app --image=docker.io/openshift/hello-openshift

3. Create a route for the application specifying your custom domain name:

   Example

   $ oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls \
   --hostname hello-openshift.${DOMAIN}

4. Label the route to admit it to your custom ingress controller:

   $ oc -n hello-world label route.route.openshift.io/hello-openshift-tls route=waf

1. Test that the app is accessible behind Amazon CloudFront:

   Example

   $ curl "https://hello-openshift.${DOMAIN}"

   Example output

   Hello OpenShift!

2. Test that the WAF denies a bad request:

   Example

   $ curl -X POST "https://hello-openshift.${DOMAIN}" \
     -F "user='<script><alert>Hello></alert></script>'"

   Example output

   <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
   <HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
   <TITLE>ERROR: The request could not be satisfied</TITLE>
   </HEAD><BODY>
   <H1>403 ERROR</H1>
   <H2>The request could not be satisfied.</H2>
   <HR noshade size="1px">
   Request blocked.
   We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
   <BR clear="all">
   If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
   <BR clear="all">
   <HR noshade size="1px">
   <PRE>
   Generated by cloudfront (CloudFront)
   Request ID: nFk9q2yB8jddI6FZOTjdliexzx-FwZtr8xUQUNT75HThPlrALDxbag==
   </PRE>
   <ADDRESS>
   </ADDRESS>
   </BODY></HTML>

   The expected result is a 403 ERROR, which means the AWS WAF is protecting your application.

1. Create a new project to deploy the AWS Load Balancer Operator into by running the following command:

   $ oc new-project aws-load-balancer-operator

2. Create an AWS IAM policy for the AWS Load Balancer Controller if one does not already exist by running the following command:

   Note

   The policy is sourced from the upstream AWS Load Balancer Controller policy. This is required by the operator to function.

   $ POLICY_ARN=$(aws iam list-policies --query \
        "Policies[?PolicyName=='aws-load-balancer-operator-policy'].{ARN:Arn}" \
        --output text)

   $ if [[ -z "${POLICY_ARN}" ]]; then
       wget -O "${SCRATCH}/load-balancer-operator-policy.json" \
          https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json
        POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
        --output text iam create-policy \
        --policy-name aws-load-balancer-operator-policy \
        --policy-document "file://${SCRATCH}/load-balancer-operator-policy.json")
   fi

3. Create an AWS IAM trust policy for AWS Load Balancer Operator:

   $ cat <<EOF > "${SCRATCH}/trust-policy.json"
   {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Condition": {
      "StringEquals" : {
        "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-operator-controller-manager", "system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-cluster"]
      }
    },
    "Principal": {
      "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity"
    }
    ]
   }
   EOF

4. Create an AWS IAM role for the AWS Load Balancer Operator:

   $ ROLE_ARN=$(aws iam create-role --role-name "${CLUSTER}-alb-operator" \
      --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
      --query Role.Arn --output text)

5. Attach the AWS Load Balancer Operator policy to the IAM role we created previously by running the following command:

   $ aws iam attach-role-policy --role-name "${CLUSTER}-alb-operator" \
        --policy-arn ${POLICY_ARN}

6. Create a secret for the AWS Load Balancer Operator to assume our newly created AWS IAM role:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   stringData:
     credentials: |
       [default]
       role_arn = ${ROLE_ARN}
       web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
   EOF

7. Install the AWS Load Balancer Operator:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     upgradeStrategy: Default
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     channel: stable-v1.0
     installPlanApproval: Automatic
     name: aws-load-balancer-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: aws-load-balancer-operator.v1.0.0
   EOF

8. Deploy an instance of the AWS Load Balancer Controller using the operator:

   Note

   If you get an error here wait a minute and try again, it means the Operator has not completed installing yet.

   $ cat << EOF | oc apply -f -
   apiVersion: networking.olm.openshift.io/v1
   kind: AWSLoadBalancerController
   metadata:
     name: cluster
   spec:
     credentials:
       name: aws-load-balancer-operator
     enabledAddons:
       - AWSWAFv2
   EOF

9. Check the that the operator and controller pods are both running:

   $ oc -n aws-load-balancer-operator get pods

   You should see the following, if not wait a moment and retry:

   NAME                                                             READY   STATUS    RESTARTS   AGE
   aws-load-balancer-controller-cluster-6ddf658785-pdp5d            1/1     Running   0          99s
   aws-load-balancer-operator-controller-manager-577d9ffcb9-w6zqn   2/2     Running   0          2m4s

1. Create a new project for our sample application:

   $ oc new-project hello-world

2. Deploy a hello world application:

   $ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift

3. Convert the pre-created service resource to a NodePort service type:

   $ oc -n hello-world patch service hello-openshift -p '{"spec":{"type":"NodePort"}}'

4. Deploy an AWS ALB using the AWS Load Balancer Operator:

   $ cat << EOF | oc apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hello-openshift-alb
     namespace: hello-world
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
   spec:
     ingressClassName: alb
     rules:
       - http:
           paths:
             - path: /
               pathType: Exact
               backend:
                 service:
                   name: hello-openshift
                   port:
                     number: 8080
   EOF

5. Curl the AWS ALB Ingress endpoint to verify the hello world application is accessible:

   Note

   AWS ALB provisioning takes a few minutes. If you receive an error that says curl: (6) Could not resolve host, please wait and try again.

   $ INGRESS=$(oc -n hello-world get ingress hello-openshift-alb -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   $ curl "http://${INGRESS}"

   Example output

   Hello OpenShift!

1. Create an IAM Policy to allow for S3 Access:

   $ POLICY_ARN=$(aws iam list-policies --query "Policies[?PolicyName=='RosaOadpVer1'].{ARN:Arn}" --output text)
   if [[ -z "${POLICY_ARN}" ]]; then
   $ cat << EOF > ${SCRATCH}/policy.json
   {
   "Version": "2012-10-17",
   "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:DeleteBucket",
        "s3:PutBucketTagging",
        "s3:GetBucketTagging",
        "s3:PutEncryptionConfiguration",
        "s3:GetEncryptionConfiguration",
        "s3:PutLifecycleConfiguration",
        "s3:GetLifecycleConfiguration",
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucketMultipartUploads",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts",
        "ec2:DescribeSnapshots",
        "ec2:DescribeVolumes",
        "ec2:DescribeVolumeAttribute",
        "ec2:DescribeVolumesModifications",
        "ec2:DescribeVolumeStatus",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:CreateSnapshot",
        "ec2:DeleteSnapshot"
      ],
      "Resource": "*"
    }
   ]}
   EOF
   $ POLICY_ARN=$(aws iam create-policy --policy-name "RosaOadpVer1" \
   --policy-document file:///${SCRATCH}/policy.json --query Policy.Arn \
   --tags Key=rosa_openshift_version,Value=${CLUSTER_VERSION} Key=rosa_role_prefix,Value=ManagedOpenShift Key=operator_namespace,Value=openshift-oadp Key=operator_name,Value=openshift-oadp \
   --output text)
   fi
   $ echo ${POLICY_ARN}

2. Create an IAM Role trust policy for the cluster:

   $ cat <<EOF > ${SCRATCH}/trust-policy.json
   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Principal": {
         "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT}"
       },
       "Action": "sts:AssumeRoleWithWebIdentity",
       "Condition": {
         "StringEquals": {
            "${OIDC_ENDPOINT}:sub": [
              "system:serviceaccount:openshift-adp:openshift-adp-controller-manager",
              "system:serviceaccount:openshift-adp:velero"]
         }
       }
     }]
   }
   EOF
   $ ROLE_ARN=$(aws iam create-role --role-name \
    "${ROLE_NAME}" \
     --assume-role-policy-document file://${SCRATCH}/trust-policy.json \
     --tags Key=rosa_cluster_id,Value=${ROSA_CLUSTER_ID} Key=rosa_openshift_version,Value=${CLUSTER_VERSION} Key=rosa_role_prefix,Value=ManagedOpenShift Key=operator_namespace,Value=openshift-adp Key=operator_name,Value=openshift-oadp \
     --query Role.Arn --output text)
   
   $ echo ${ROLE_ARN}

3. Attach the IAM Policy to the IAM Role:

   $ aws iam attach-role-policy --role-name "${ROLE_NAME}" \
    --policy-arn ${POLICY_ARN}

1. Create a namespace for OADP:

   $ oc create namespace openshift-adp

2. Create a credentials secret:

   $ cat <<EOF > ${SCRATCH}/credentials
   [default]
   role_arn = ${ROLE_ARN}
   web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
   EOF
   $ oc -n openshift-adp create secret generic cloud-credentials \
    --from-file=${SCRATCH}/credentials

3. Deploy the OADP Operator:

   Note

   There is currently an issue with version 1.1 of the Operator with backups that have a PartiallyFailed status. This does not seem to affect the backup and restore process, but it should be noted as there are issues with it.

   $ cat << EOF | oc create -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
    generateName: openshift-adp-
    namespace: openshift-adp
    name: oadp
   spec:
    targetNamespaces:
    - openshift-adp
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
    name: redhat-oadp-operator
    namespace: openshift-adp
   spec:
    channel: stable-1.2
    installPlanApproval: Automatic
    name: redhat-oadp-operator
    source: redhat-operators
    sourceNamespace: openshift-marketplace
   EOF

4. Wait for the Operator to be ready:

   $ watch oc -n openshift-adp get pods

   Example output

   NAME                                                READY   STATUS    RESTARTS   AGE
   openshift-adp-controller-manager-546684844f-qqjhn   1/1     Running   0          22s

5. Create Cloud Storage:

   $ cat << EOF | oc create -f -
   apiVersion: oadp.openshift.io/v1alpha1
   kind: CloudStorage
   metadata:
    name: ${CLUSTER_NAME}-oadp
    namespace: openshift-adp
   spec:
    creationSecret:
      key: credentials
      name: cloud-credentials
    enableSharedConfig: true
    name: ${CLUSTER_NAME}-oadp
    provider: aws
    region: $REGION
   EOF

6. Check your application’s storage default storage class:

   $ oc get pvc -n <namespace> 1

   1

   Enter your application’s namespace.

   Example output

   NAME     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   applog   Bound    pvc-351791ae-b6ab-4e8b-88a4-30f73caf5ef8   1Gi        RWO            gp3-csi        4d19h
   mysql    Bound    pvc-16b8e009-a20a-4379-accc-bc81fedd0621   1Gi        RWO            gp3-csi        4d19h

   $ oc get storageclass

   Example output

   NAME                PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   gp2                 kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   true                   4d21h
   gp2-csi             ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h
   gp3                 ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h
   gp3-csi (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   4d21h

   Using either gp3-csi, gp2-csi, gp3 or gp2 will work. If the application(s) that are being backed up are all using PV’s with CSI, include the CSI plugin in the OADP DPA configuration.

7. CSI only: Deploy a Data Protection Application:

   $ cat << EOF | oc create -f -
   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
    name: ${CLUSTER_NAME}-dpa
    namespace: openshift-adp
   spec:
    backupImages: true
    features:
      dataMover:
        enable: false
    backupLocations:
    - bucket:
        cloudStorageRef:
          name: ${CLUSTER_NAME}-oadp
        credential:
          key: credentials
          name: cloud-credentials
        prefix: velero
        default: true
        config:
          region: ${REGION}
    configuration:
      velero:
        defaultPlugins:
        - openshift
        - aws
        - csi
      restic:
        enable: false
   EOF

   Note

   If you run this command for CSI volumes, you can skip the next step.

8. Non-CSI volumes: Deploy a Data Protection Application:

   $ cat << EOF | oc create -f -
   apiVersion: oadp.openshift.io/v1alpha1
   kind: DataProtectionApplication
   metadata:
    name: ${CLUSTER_NAME}-dpa
    namespace: openshift-adp
   spec:
    backupImages: true
    features:
      dataMover:
        enable: false
    backupLocations:
    - bucket:
        cloudStorageRef:
          name: ${CLUSTER_NAME}-oadp
        credential:
          key: credentials
          name: cloud-credentials
        prefix: velero
        default: true
        config:
          region: ${REGION}
    configuration:
      velero:
        defaultPlugins:
        - openshift
        - aws
      restic:
        enable: false
    snapshotLocations:
      - velero:
          config:
            credentialsFile: /tmp/credentials/openshift-adp/cloud-credentials-credentials
            enableSharedConfig: 'true'
            profile: default
            region: ${REGION}
          provider: aws
   EOF

1. Create a workload to back up:

   $ oc create namespace hello-world
   $ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift

2. Expose the route:

   $ oc expose service/hello-openshift -n hello-world

3. Check that the application is working:

   $ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`

   Example output

   Hello OpenShift!

4. Back up the workload:

   $ cat << EOF | oc create -f -
   apiVersion: velero.io/v1
   kind: Backup
   metadata:
    name: hello-world
    namespace: openshift-adp
   spec:
    includedNamespaces:
    - hello-world
    storageLocation: ${CLUSTER_NAME}-dpa-1
    ttl: 720h0m0s
   EOF

5. Wait until the backup is done:

   $ watch "oc -n openshift-adp get backup hello-world -o json | jq .status"

   Example output

   {
    "completionTimestamp": "2022-09-07T22:20:44Z",
    "expiration": "2022-10-07T22:20:22Z",
    "formatVersion": "1.1.0",
    "phase": "Completed",
    "progress": {
      "itemsBackedUp": 58,
      "totalItems": 58
    },
    "startTimestamp": "2022-09-07T22:20:22Z",
    "version": 1
   }

6. Delete the demo workload:

   $ oc delete ns hello-world

7. Restore from the backup:

   $ cat << EOF | oc create -f -
   apiVersion: velero.io/v1
   kind: Restore
   metadata:
    name: hello-world
    namespace: openshift-adp
   spec:
    backupName: hello-world
   EOF

8. Wait for the Restore to finish:

   $ watch "oc -n openshift-adp get restore hello-world -o json | jq .status"

   Example output

   {
    "completionTimestamp": "2022-09-07T22:25:47Z",
    "phase": "Completed",
    "progress": {
      "itemsRestored": 38,
      "totalItems": 38
    },
    "startTimestamp": "2022-09-07T22:25:28Z",
    "warnings": 9
   }

9. Check that the workload is restored:

   $ oc -n hello-world get pods

   Example output

   NAME                              READY   STATUS    RESTARTS   AGE
   hello-openshift-9f885f7c6-kdjpj   1/1     Running   0          90s

   $ curl `oc get route/hello-openshift -n hello-world -o jsonpath='{.spec.host}'`

   Example output

   Hello OpenShift!

10. For troubleshooting tips please refer to the OADP team’s troubleshooting documentation
11. Additional sample applications can be found in the OADP team’s sample applications directory

1. Delete the workload:

   $ oc delete ns hello-world

2. Remove the backup and restore resources from the cluster if they are no longer required:

   $ oc delete backup hello-world
   $ oc delete restore hello-world

3. To delete the backup/restore and remote objects in s3:

   $ velero backup delete hello-world
   $ velero restore delete hello-world

4. Delete the Data Protection Application:

   $ oc -n openshift-adp delete dpa ${CLUSTER_NAME}-dpa

5. Delete the Cloud Storage:

   $ oc -n openshift-adp delete cloudstorage ${CLUSTER_NAME}-oadp

   Warning

   If this command hangs, you might need to delete the finalizer:

   $ oc -n openshift-adp patch cloudstorage ${CLUSTER_NAME}-oadp -p '{"metadata":{"finalizers":null}}' --type=merge

6. Remove the Operator if it is no longer required:

   $ oc -n openshift-adp delete subscription oadp-operator

7. Remove the namespace for the Operator:

   $ oc delete ns redhat-openshift-adp

8. Remove the Custom Resource Definitions from the cluster if you no longer wish to have them:

   $ for CRD in `oc get crds | grep velero | awk '{print $1}'`; do oc delete crd $CRD; done
   $ for CRD in `oc get crds | grep -i oadp | awk '{print $1}'`; do oc delete crd $CRD; done

9. Delete the AWS S3 Bucket:

   $ aws s3 rm s3://${CLUSTER_NAME}-oadp --recursive
   $ aws s3api delete-bucket --bucket ${CLUSTER_NAME}-oadp

10. Detach the Policy from the role:

    $ aws iam detach-role-policy --role-name "${ROLE_NAME}" \
     --policy-arn "${POLICY_ARN}"

11. Delete the role:

    $ aws iam delete-role --role-name "${ROLE_NAME}"

1. Create an AWS IAM policy for the AWS Load Balancer Controller:

   Note

   The policy is sourced from the upstream AWS Load Balancer Controller policy plus permission to create tags on subnets. This is required by the operator to function.

   $ oc new-project aws-load-balancer-operator
   $ POLICY_ARN=$(aws iam list-policies --query \
        "Policies[?PolicyName=='aws-load-balancer-operator-policy'].{ARN:Arn}" \
        --output text)
   $ if [[ -z "${POLICY_ARN}" ]]; then
       wget -O "${SCRATCH}/load-balancer-operator-policy.json" \
          https://raw.githubusercontent.com/rh-mobb/documentation/main/content/rosa/aws-load-balancer-operator/load-balancer-operator-policy.json
        POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
        --output text iam create-policy \
        --policy-name aws-load-balancer-operator-policy \
        --policy-document "file://${SCRATCH}/load-balancer-operator-policy.json")
   fi
   $ echo $POLICY_ARN

2. Create an AWS IAM trust policy for AWS Load Balancer Operator:

   $ cat <<EOF > "${SCRATCH}/trust-policy.json"
   {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Condition": {
      "StringEquals" : {
        "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-operator-controller-manager", "system:serviceaccount:aws-load-balancer-operator:aws-load-balancer-controller-cluster"]
      }
    },
    "Principal": {
      "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity"
    }
    ]
   }
   EOF

3. Create an AWS IAM role for the AWS Load Balancer Operator:

   $ ROLE_ARN=$(aws iam create-role --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
      --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
      --query Role.Arn --output text)
   $ echo $ROLE_ARN
   
   $ aws iam attach-role-policy --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
        --policy-arn $POLICY_ARN

4. Create a secret for the AWS Load Balancer Operator to assume our newly created AWS IAM role:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   stringData:
     credentials: |
       [default]
       role_arn = $ROLE_ARN
       web_identity_token_file = /var/run/secrets/openshift/serviceaccount/token
   EOF

5. Install the AWS Load Balancer Operator:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     upgradeStrategy: Default
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: aws-load-balancer-operator
     namespace: aws-load-balancer-operator
   spec:
     channel: stable-v1.0
     installPlanApproval: Automatic
     name: aws-load-balancer-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     startingCSV: aws-load-balancer-operator.v1.0.0
   EOF

6. Deploy an instance of the AWS Load Balancer Controller using the operator:

   Note

   If you get an error here wait a minute and try again, it means the Operator has not completed installing yet.

   $ cat << EOF | oc apply -f -
   apiVersion: networking.olm.openshift.io/v1
   kind: AWSLoadBalancerController
   metadata:
     name: cluster
   spec:
     credentials:
       name: aws-load-balancer-operator
   EOF

7. Check the that the operator and controller pods are both running:

   $ oc -n aws-load-balancer-operator get pods

   You should see the following, if not wait a moment and retry:

   NAME                                                             READY   STATUS    RESTARTS   AGE
   aws-load-balancer-controller-cluster-6ddf658785-pdp5d            1/1     Running   0          99s
   aws-load-balancer-operator-controller-manager-577d9ffcb9-w6zqn   2/2     Running   0          2m4s

1. Create a new project:

   $ oc new-project hello-world

2. Deploy a hello world application:

   $ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift

3. Configure a NodePort service for the AWS ALB to connect to:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Service
   metadata:
     name: hello-openshift-nodeport
     namespace: hello-world
   spec:
     ports:
       - port: 80
         targetPort: 8080
         protocol: TCP
     type: NodePort
     selector:
       deployment: hello-openshift
   EOF

4. Deploy an AWS ALB using the AWS Load Balancer Operator:

   $ cat << EOF | oc apply -f -
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hello-openshift-alb
     namespace: hello-world
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
   spec:
     ingressClassName: alb
     rules:
       - http:
           paths:
             - path: /
               pathType: Exact
               backend:
                 service:
                   name: hello-openshift-nodeport
                   port:
                     number: 80
   EOF

5. Curl the AWS ALB Ingress endpoint to verify the hello world application is accessible:

   Note

   AWS ALB provisioning takes a few minutes. If you receive an error that says curl: (6) Could not resolve host, please wait and try again.

   $ INGRESS=$(oc -n hello-world get ingress hello-openshift-alb \
       -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   $ curl "http://${INGRESS}"

   Example output

   Hello OpenShift!

6. Deploy an AWS NLB for your hello world application:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Service
   metadata:
     name: hello-openshift-nlb
     namespace: hello-world
     annotations:
       service.beta.kubernetes.io/aws-load-balancer-type: external
       service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
       service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
   spec:
     ports:
       - port: 80
         targetPort: 8080
         protocol: TCP
     type: LoadBalancer
     selector:
       deployment: hello-openshift
   EOF

7. Test the AWS NLB endpoint:

   Note

   NLB provisioning takes a few minutes. If you receive an error that says curl: (6) Could not resolve host, please wait and try again.

   $ NLB=$(oc -n hello-world get service hello-openshift-nlb \
     -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
   $ curl "http://${NLB}"

   Example output

   Hello OpenShift!

1. Delete the hello world application namespace (and all the resources in the namespace):

   $ oc delete project hello-world

2. Delete the AWS Load Balancer Operator and the AWS IAM roles:

   $ oc delete subscription aws-load-balancer-operator -n aws-load-balancer-operator
   $ aws iam detach-role-policy \
     --role-name "${ROSA_CLUSTER_NAME}-alb-operator" \
     --policy-arn $POLICY_ARN
   $ aws iam delete-role \
     --role-name "${ROSA_CLUSTER_NAME}-alb-operator"

3. Delete the AWS IAM policy:

   $ aws iam delete-policy --policy-arn $POLICY_ARN

Procedure

1. Create the cluster’s OAuth callback URL by changing the specified variables and running the following command:

   Note

   Remember to save this callback URL; it will be required later in the process.

   $ domain=$(rosa describe cluster -c <cluster_name> | grep "DNS" | grep -oE '\S+.openshiftapps.com')
   $ echo "OAuth callback URL: https://oauth-openshift.apps.$domain/oauth2callback/AAD"

   The "AAD" directory at the end of the OAuth callback URL must match the OAuth identity provider name that you will set up later in this process.

2. Create the Entra ID application by logging in to the Azure portal, and select the App registrations blade. Then, select New registration to create a new application.

   

3. Name the application, for example openshift-auth.
4. Select Web from the Redirect URI dropdown and enter the value of the OAuth callback URL you retrieved in the previous step.

5. After providing the required information, click Register to create the application.

   

6. Select the Certificates & secrets sub-blade and select New client secret.

   

7. Complete the requested details and store the generated client secret value. This secret is required later in this process.

   Important

   After initial setup, you cannot see the client secret. If you did not record the client secret, you must generate a new one.

   

8. Select the Overview sub-blade and note the Application (client) ID and Directory (tenant) ID. You will need these values in a future step.

   

1. Click the Token configuration sub-blade and select the Add optional claim button.

   

2. Select the ID radio button.

   

3. Select the email claim checkbox.

   

4. Select the preferred_username claim checkbox. Then, click Add to configure the email and preferred_username claims your Entra ID application.

   

5. A dialog box appears at the top of the page. Follow the prompt to enable the necessary Microsoft Graph permissions.

   

Procedure

1. From the Token configuration sub-blade, click Add groups claim.

   

2. To configure group claims for your Entra ID application, select Security groups and then click the Add.

   Note

   In this example, the group claim includes all of the security groups that a user is a member of. In a real production environment, ensure that the groups that the group claim only includes groups that apply to Red Hat OpenShift Service on AWS.

   

Procedure

1. Create the variables by running the following command:

   $ CLUSTER_NAME=example-cluster 1
   $ IDP_NAME=AAD 2
   $ APP_ID=yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy 3
   $ CLIENT_SECRET=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx 4
   $ TENANT_ID=zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz 5

   1

   Replace this with the name of your ROSA cluster.

   2

   Replace this value with the name you used in the OAuth callback URL that you generated earlier in this process.

   3

   Replace this with the Application (client) ID.

   4

   Replace this with the Client Secret.

   5

   Replace this with the Directory (tenant) ID.

2. Configure the cluster’s OAuth provider by running the following command. If you enabled group claims, ensure that you use the --group-claims groups argument.

   • If you enabled group claims, run the following command:

     $ rosa create idp \
     --cluster ${CLUSTER_NAME} \
     --type openid \
     --name ${IDP_NAME} \
     --client-id ${APP_ID} \
     --client-secret ${CLIENT_SECRET} \
     --issuer-url https://login.microsoftonline.com/${TENANT_ID}/v2.0 \
     --email-claims email \
     --name-claims name \
     --username-claims preferred_username \
     --extra-scopes email,profile \
     --groups-claims groups

   • If you did not enable group claims, run the following command:

     $ rosa create idp \
     --cluster ${CLUSTER_NAME} \
     --type openid \
     --name ${IDP_NAME} \
     --client-id ${APP_ID} \
     --client-secret ${CLIENT_SECRET} \
     --issuer-url https://login.microsoftonline.com/${TENANT_ID}/v2.0 \
     --email-claims email \
     --name-claims name \
     --username-claims preferred_username \
     --extra-scopes email,profile

1. Log in to your ROSA cluster by running the following command:

   $ oc login --token=<your-token> --server=<your-server-url>

   You can find your login token by accessing your cluster in pull secret from Red Hat OpenShift Cluster Manager.

2. Validate that your cluster has STS by running the following command:

   $ oc get authentication.config.openshift.io cluster -o json \
     | jq .spec.serviceAccountIssuer

   Example output

   "https://xxxxx.cloudfront.net/xxxxx"

   If your output is different, do not proceed. See Red Hat documentation on creating an STS cluster before continuing this process.

3. Set the SecurityContextConstraints permission to allow the CSI driver to run by running the following command:

   $ oc new-project csi-secrets-store
   $ oc adm policy add-scc-to-user privileged \
       system:serviceaccount:csi-secrets-store:secrets-store-csi-driver
   $ oc adm policy add-scc-to-user privileged \
       system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws

4. Create environment variables to use later in this process by running the following command:

   $ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
   $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster \
      -o jsonpath='{.spec.serviceAccountIssuer}' | sed  's|^https://||')
   $ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
   $ export AWS_PAGER=""

1. Use Helm to register the secrets store CSI driver by running the following command:

   $ helm repo add secrets-store-csi-driver \
       https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts

2. Update your Helm repositories by running the following command:

   $ helm repo update

3. Install the secrets store CSI driver by running the following command:

   $ helm upgrade --install -n csi-secrets-store \
       csi-secrets-store-driver secrets-store-csi-driver/secrets-store-csi-driver

4. Deploy the AWS provider by running the following command:

   $ oc -n csi-secrets-store apply -f \
       https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml

5. Check that both Daemonsets are running by running the following command:

   $ oc -n csi-secrets-store get ds \
       csi-secrets-store-provider-aws \
       csi-secrets-store-driver-secrets-store-csi-driver

6. Label the Secrets Store CSI Driver to allow use with the restricted pod security profile by running the following command:

   $ oc label csidriver.storage.k8s.io/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted

1. Create a secret in Secrets Manager by running the following command:

   $ SECRET_ARN=$(aws --region "$REGION" secretsmanager create-secret \
       --name MySecret --secret-string \
       '{"username":"shadowman", "password":"hunter2"}' \
       --query ARN --output text); echo $SECRET_ARN

2. Create an IAM Access Policy document by running the following command:

   $ cat << EOF > policy.json
   {
      "Version": "2012-10-17",
      "Statement": [{
         "Effect": "Allow",
         "Action": [
           "secretsmanager:GetSecretValue",
           "secretsmanager:DescribeSecret"
         ],
         "Resource": ["$SECRET_ARN"]
         }]
   }
   EOF

3. Create an IAM Access Policy by running the following command:

   $ POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn \
   --output text iam create-policy \
   --policy-name openshift-access-to-mysecret-policy \
   --policy-document file://policy.json); echo $POLICY_ARN

4. Create an IAM Role trust policy document by running the following command:

   Note

   The trust policy is locked down to the default service account of a namespace you create later in this process.

   $ cat <<EOF > trust-policy.json
   {
      "Version": "2012-10-17",
      "Statement": [
      {
      "Effect": "Allow",
      "Condition": {
        "StringEquals" : {
          "${OIDC_ENDPOINT}:sub": ["system:serviceaccount:my-application:default"]
         }
       },
       "Principal": {
          "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
       },
       "Action": "sts:AssumeRoleWithWebIdentity"
       }
       ]
   }
   EOF

5. Create an IAM role by running the following command:

   $ ROLE_ARN=$(aws iam create-role --role-name openshift-access-to-mysecret \
   --assume-role-policy-document file://trust-policy.json \
   --query Role.Arn --output text); echo $ROLE_ARN

6. Attach the role to the policy by running the following command:

   $ aws iam attach-role-policy --role-name openshift-access-to-mysecret \
       --policy-arn $POLICY_ARN

1. Create an OpenShift project by running the following command:

   $ oc new-project my-application

2. Annotate the default service account to use the STS Role by running the following command:

   $ oc annotate -n my-application serviceaccount default \
       eks.amazonaws.com/role-arn=$ROLE_ARN

3. Create a secret provider class to access our secret by running the following command:

   $ cat << EOF | oc apply -f -
   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: my-application-aws-secrets
   spec:
     provider: aws
     parameters:
       objects: |
         - objectName: "MySecret"
           objectType: "secretsmanager"
   EOF

4. Create a deployment by using our secret in the following command:

   $ cat << EOF | oc apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: my-application
     labels:
       app: my-application
   spec:
     volumes:
     - name: secrets-store-inline
       csi:
         driver: secrets-store.csi.k8s.io
         readOnly: true
         volumeAttributes:
           secretProviderClass: "my-application-aws-secrets"
     containers:
     - name: my-application-deployment
       image: k8s.gcr.io/e2e-test-images/busybox:1.29
       command:
         - "/bin/sleep"
         - "10000"
       volumeMounts:
       - name: secrets-store-inline
         mountPath: "/mnt/secrets-store"
         readOnly: true
   EOF

5. Verify the pod has the secret mounted by running the following command:

   $ oc exec -it my-application -- cat /mnt/secrets-store/MySecret

1. Delete the application by running the following command:

   $ oc delete project my-application

2. Delete the secrets store csi driver by running the following command:

   $ helm delete -n csi-secrets-store csi-secrets-store-driver

3. Delete the security context constraints by running the following command:

   $ oc adm policy remove-scc-from-user privileged \
       system:serviceaccount:csi-secrets-store:secrets-store-csi-driver; oc adm policy remove-scc-from-user privileged \
       system:serviceaccount:csi-secrets-store:csi-secrets-store-provider-aws

4. Delete the AWS provider by running the following command:

   $ oc -n csi-secrets-store delete -f \
   https://raw.githubusercontent.com/rh-mobb/documentation/main/content/misc/secrets-store-csi/aws-provider-installer.yaml

5. Delete AWS Roles and Policies by running the following command:

   $ aws iam detach-role-policy --role-name openshift-access-to-mysecret \
       --policy-arn $POLICY_ARN; aws iam delete-role --role-name openshift-access-to-mysecret; aws iam delete-policy --policy-arn $POLICY_ARN

6. Delete the Secrets Manager secret by running the following command:

   $ aws secretsmanager --region $REGION delete-secret --secret-id $SECRET_ARN

1. Configure the following environment variables, changing the cluster name to suit your cluster:

   $ export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
   $ export REGION=$(rosa describe cluster -c ${ROSA_CLUSTER_NAME} --output json | jq -r .region.id)
   $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
   $ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --query Account --output text`
   $ export ACK_SERVICE=s3
   $ export ACK_SERVICE_ACCOUNT=ack-${ACK_SERVICE}-controller
   $ export POLICY_ARN=arn:aws:iam::aws:policy/AmazonS3FullAccess
   $ export AWS_PAGER=""
   $ export SCRATCH="/tmp/${ROSA_CLUSTER_NAME}/ack"
   $ mkdir -p ${SCRATCH}

2. Ensure all fields output correctly before moving to the next section:

   $ echo "Cluster: ${ROSA_CLUSTER_NAME}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

1. Create an AWS Identity Access Management (IAM) trust policy for the ACK Operator:

   $ cat <<EOF > "${SCRATCH}/trust-policy.json"
   {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Condition": {
      "StringEquals" : {
        "${OIDC_ENDPOINT}:sub": "system:serviceaccount:ack-system:${ACK_SERVICE_ACCOUNT}"
      }
    },
    "Principal": {
      "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity"
    }
    ]
   }
   EOF

2. Create an AWS IAM role for the ACK Operator to assume with the AmazonS3FullAccess policy attached:

   Note

   You can find the recommended policy in each project’s GitHub repository, for example https://github.com/aws-controllers-k8s/s3-controller/blob/main/config/iam/recommended-policy-arn.

   $ ROLE_ARN=$(aws iam create-role --role-name "ack-${ACK_SERVICE}-controller" \
      --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
      --query Role.Arn --output text)
   $ echo $ROLE_ARN
   
   $ aws iam attach-role-policy --role-name "ack-${ACK_SERVICE}-controller" \
        --policy-arn ${POLICY_ARN}

1. Create a project to install the ACK S3 Operator into:

   $ oc new-project ack-system

2. Create a file with the ACK S3 Operator configuration:

   Note

   ACK_WATCH_NAMESPACE is purposefully left blank so the controller can properly watch all namespaces in the cluster.

   $ cat <<EOF > "${SCRATCH}/config.txt"
   ACK_ENABLE_DEVELOPMENT_LOGGING=true
   ACK_LOG_LEVEL=debug
   ACK_WATCH_NAMESPACE=
   AWS_REGION=${REGION}
   AWS_ENDPOINT_URL=
   ACK_RESOURCE_TAGS=${CLUSTER_NAME}
   ENABLE_LEADER_ELECTION=true
   LEADER_ELECTION_NAMESPACE=
   EOF

3. Use the file from the previous step to create a ConfigMap:

   $ oc -n ack-system create configmap \
     --from-env-file=${SCRATCH}/config.txt ack-${ACK_SERVICE}-user-config

4. Install the ACK S3 Operator from OperatorHub:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: ack-${ACK_SERVICE}-controller
     namespace: ack-system
   spec:
     upgradeStrategy: Default
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: ack-${ACK_SERVICE}-controller
     namespace: ack-system
   spec:
     channel: alpha
     installPlanApproval: Automatic
     name: ack-${ACK_SERVICE}-controller
     source: community-operators
     sourceNamespace: openshift-marketplace
   EOF

5. Annotate the ACK S3 Operator service account with the AWS IAM role to assume and restart the deployment:

   $ oc -n ack-system annotate serviceaccount ${ACK_SERVICE_ACCOUNT} \
     eks.amazonaws.com/role-arn=${ROLE_ARN} && \
     oc -n ack-system rollout restart deployment ack-${ACK_SERVICE}-controller

6. Verify that the ACK S3 Operator is running:

   $ oc -n ack-system get pods

   Example output

   NAME                                 READY   STATUS    RESTARTS   AGE
   ack-s3-controller-585f6775db-s4lfz   1/1     Running   0          51s

1. Deploy an S3 bucket resource:

   $ cat << EOF | oc apply -f -
   apiVersion: s3.services.k8s.aws/v1alpha1
   kind: Bucket
   metadata:
      name: ${CLUSTER-NAME}-bucket
      namespace: ack-system
   spec:
      name: ${CLUSTER-NAME}-bucket
   EOF

2. Verify the S3 bucket was created in AWS:

   $ aws s3 ls | grep ${CLUSTER_NAME}-bucket

   Example output

   2023-10-04 14:51:45 mrmc-test-maz-bucket

1. Delete the S3 bucket resource:

   $ oc -n ack-system delete bucket.s3.services.k8s.aws/${CLUSTER-NAME}-bucket

2. Delete the ACK S3 Operator and the AWS IAM roles:

   $ oc -n ack-system delete subscription ack-${ACK_SERVICE}-controller
   $ aws iam detach-role-policy \
     --role-name "ack-${ACK_SERVICE}-controller" \
     --policy-arn ${POLICY_ARN}
   $ aws iam delete-role \
     --role-name "ack-${ACK_SERVICE}-controller"

3. Delete the ack-system project:

   $ oc delete project ack-system

1. Configure the following environment variables:

   $ export DOMAIN=<apps.example.com> 1
   $ export AWS_PAGER=""
   $ export CLUSTER=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
   $ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
   $ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
   $ export SCRATCH="/tmp/${CLUSTER}/external-dns"
   $ mkdir -p ${SCRATCH}

   1

   Replace with the custom domain you want to use for the IngressController.

2. Ensure all fields output correctly before moving to the next section:

   $ echo "Cluster: ${CLUSTER}, Region: ${REGION}, AWS Account ID: ${AWS_ACCOUNT_ID}"

   Note

   The "Cluster" output from the previous command may be the name of your cluster, the internal ID of your cluster, or the cluster’s domain prefix. If you prefer to use another identifier, you can manually set this value by running the following command:

   $ export CLUSTER=my-custom-value

Procedure

1. Create a new TLS secret from a private key and a public certificate, where fullchain.pem is your full wildcard certificate chain (including any intermediaries) and privkey.pem is your wildcard certificate’s private key:

   $ oc -n openshift-ingress create secret tls external-dns-tls --cert=fullchain.pem --key=privkey.pem

2. Create a new IngressController resource:

   $ cat << EOF | oc apply -f -
   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: external-dns-ingress
     namespace: openshift-ingress-operator
   spec:
     domain: ${DOMAIN}
     defaultCertificate:
       name: external-dns-tls
     endpointPublishingStrategy:
       loadBalancer:
         dnsManagementPolicy: Unmanaged
         providerParameters:
           aws:
             type: NLB
           type: AWS
         scope: External
       type: LoadBalancerService
   EOF

   Warning

   This IngressController example will create an internet accessible Network Load Balancer (NLB) in your AWS account. To provision an internal NLB instead, set the .spec.endpointPublishingStrategy.loadBalancer.scope parameter to Internal before creating the IngressController resource.

3. Verify that your custom domain IngressController has successfully created an external load balancer:

   $ oc -n openshift-ingress get service/router-external-dns-ingress

   Example output

   NAME                          TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)                      AGE
   router-external-dns-ingress   LoadBalancer   172.30.71.250   a4838bb991c6748439134ab89f132a43-aeae124077b50c01.elb.us-east-1.amazonaws.com   80:32227/TCP,443:30310/TCP   43s

1. Retrieve the Amazon Route 53 public hosted zone ID:

   $ export ZONE_ID=$(aws route53 list-hosted-zones-by-name --output json \
     --dns-name "${DOMAIN}." --query 'HostedZones[0]'.Id --out text | sed 's/\/hostedzone\///')

2. Prepare a document with the necessary DNS changes to enable DNS resolution for the canonical domain of the Ingress Controller:

   $ NLB_HOST=$(oc -n openshift-ingress get service/router-external-dns-ingress -ojsonpath="{.status.loadBalancer.ingress[0].hostname}")
   $ cat << EOF > "${SCRATCH}/create-cname.json"
   {
     "Comment":"Add CNAME to ingress controller canonical domain",
     "Changes":[{
         "Action":"CREATE",
         "ResourceRecordSet":{
           "Name": "router-external-dns-ingress.${DOMAIN}",
         "Type":"CNAME",
         "TTL":30,
         "ResourceRecords":[{
           "Value": "${NLB_HOST}"
         }]
       }
     }]
   }
   EOF

   The External DNS Operator uses this canonical domain as the target for CNAME records.

3. Submit your changes to Amazon Route 53 for propagation:

   aws route53 change-resource-record-sets \
     --hosted-zone-id ${ZONE_ID} \
     --change-batch file://${SCRATCH}/create-cname.json

4. Create an AWS IAM Policy document that allows the External DNS Operator to update only the custom domain public hosted zone:

   $ cat << EOF > "${SCRATCH}/external-dns-policy.json"
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "route53:ChangeResourceRecordSets"
         ],
         "Resource": [
           "arn:aws:route53:::hostedzone/${ZONE_ID}"
         ]
       },
       {
         "Effect": "Allow",
         "Action": [
           "route53:ListHostedZones",
           "route53:ListResourceRecordSets"
         ],
         "Resource": [
           "*"
         ]
       }
     ]
   }
   EOF

5. Create an AWS IAM user:

   $ aws iam create-user --user-name "${CLUSTER}-external-dns-operator"

6. Attach the policy:

   $ aws iam attach-user-policy --user-name "${CLUSTER}-external-dns-operator" --policy-arn $POLICY_ARN

   Note

   This will be changed to STS using IRSA in the future.

7. Create AWS keys for the IAM user:

   $ SECRET_ACCESS_KEY=$(aws iam create-access-key --user-name "${CLUSTER}-external-dns-operator")

8. Create static credentials:

   $ cat << EOF > "${SCRATCH}/credentials"
   [default]
   aws_access_key_id = $(echo $SECRET_ACCESS_KEY | jq -r '.AccessKey.AccessKeyId')
   aws_secret_access_key = $(echo $SECRET_ACCESS_KEY | jq -r '.AccessKey.SecretAccessKey')
   EOF

1. Create a new project:

   $ oc new-project external-dns-operator

2. Install the External DNS Operator from OperatorHub:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: external-dns-group
     namespace: external-dns-operator
   spec:
     targetNamespaces:
     - external-dns-operator
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: external-dns-operator
     namespace: external-dns-operator
   spec:
     channel: stable-v1.1
     installPlanApproval: Automatic
     name: external-dns-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

3. Wait until the External DNS Operator is running:

   $ oc rollout status deploy external-dns-operator --timeout=300s

4. Create a secret from the AWS IAM user credentials:

   $ oc -n external-dns-operator create secret generic external-dns \
     --from-file "${SCRATCH}/credentials"

5. Deploy the ExternalDNS controller:

   $ cat << EOF | oc apply -f -
   apiVersion: externaldns.olm.openshift.io/v1beta1
   kind: ExternalDNS
   metadata:
     name: ${DOMAIN}
   spec:
     domains:
       - filterType: Include
         matchType: Exact
         name: ${DOMAIN}
     provider:
       aws:
         credentials:
           name: external-dns
       type: AWS
     source:
       openshiftRouteOptions:
         routerName: external-dns-ingress
       type: OpenShiftRoute
     zones:
       - ${ZONE_ID}
   EOF

6. Wait until the controller is running:

   $ oc rollout status deploy external-dns-${DOMAIN} --timeout=300s

1. Create a new project for your sample application:

   $ oc new-project hello-world

2. Deploy a hello world application:

   $ oc new-app -n hello-world --image=docker.io/openshift/hello-openshift

3. Create a route for the application specifying your custom domain name:

   $ oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls \
   --hostname hello-openshift.${DOMAIN}

4. Check if the DNS record was created automatically by ExternalDNS:

   Note

   It can take a few minutes for the record to appear in Amazon Route 53.

   $ aws route53 list-resource-record-sets --hosted-zone-id ${ZONE_ID} \
      --query "ResourceRecordSets[?Type == 'CNAME']" | grep hello-openshift

5. Optional: You can also view the TXT records that indicate they were created by ExternalDNS:

   $ aws route53 list-resource-record-sets --hosted-zone-id ${ZONE_ID} \
      --query "ResourceRecordSets[?Type == 'TXT']" | grep ${DOMAIN}

6. Curl the newly created DNS record to your sample application to verify the hello world application is accessible:

   $ curl https://hello-openshift.${DOMAIN}

   Example output

   Hello OpenShift!

1. Configure the following environment variables:

   $ export DOMAIN=apps.example.com 1
   $ export EMAIL=email@example.com 2
   $ export AWS_PAGER=""
   $ export CLUSTER=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')
   $ export OIDC_ENDPOINT=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer | sed  's|^https://||')
   $ export REGION=$(oc get infrastructure cluster -o=jsonpath="{.status.platformStatus.aws.region}")
   $ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
   $ export SCRATCH="/tmp/${CLUSTER}/dynamic-certs"
   $ mkdir -p ${SCRATCH}

   1

   Replace with the custom domain you want to use for the IngressController.

   2

   Replace with the e-mail you want Let’s Encrypt to use to send notifications about your certificates.

2. Ensure all fields output correctly before moving to the next section:

   $ echo "Cluster: ${CLUSTER}, Region: ${REGION}, OIDC Endpoint: ${OIDC_ENDPOINT}, AWS Account ID: ${AWS_ACCOUNT_ID}"

   Note

   The "Cluster" output from the previous command may be the name of your cluster, the internal ID of your cluster, or the cluster’s domain prefix. If you prefer to use another identifier, you can manually set this value by running the following command:

   $ export CLUSTER=my-custom-value

1. Retrieve the Amazon Route 53 public hosted zone ID:

   Note

   This command looks for a public hosted zone that matches the custom domain you specified earlier as the DOMAIN environment variable. You can manually specify the Amazon Route 53 public hosted zone by running export ZONE_ID=<zone_ID>, replacing <zone_ID> with your specific Amazon Route 53 public hosted zone ID.

   $ export ZONE_ID=$(aws route53 list-hosted-zones-by-name --output json \
     --dns-name "${DOMAIN}." --query 'HostedZones[0]'.Id --out text | sed 's/\/hostedzone\///')

2. Create an AWS IAM policy document for the cert-manager Operator that provides the ability to update only the specified public hosted zone:

   $ cat <<EOF > "${SCRATCH}/cert-manager-policy.json"
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": "route53:GetChange",
         "Resource": "arn:aws:route53:::change/*"
       },
       {
         "Effect": "Allow",
         "Action": [
           "route53:ChangeResourceRecordSets",
           "route53:ListResourceRecordSets"
         ],
         "Resource": "arn:aws:route53:::hostedzone/${ZONE_ID}"
       },
       {
         "Effect": "Allow",
         "Action": "route53:ListHostedZonesByName",
         "Resource": "*"
       }
     ]
   }
   EOF

3. Create the IAM policy using the file you created in the previous step:

   $ POLICY_ARN=$(aws iam create-policy --policy-name "${CLUSTER}-cert-manager-policy" \
     --policy-document file://${SCRATCH}/cert-manager-policy.json \
     --query 'Policy.Arn' --output text)

4. Create an AWS IAM trust policy for the cert-manager Operator:

   $ cat <<EOF > "${SCRATCH}/trust-policy.json"
   {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Condition": {
      "StringEquals" : {
        "${OIDC_ENDPOINT}:sub": "system:serviceaccount:cert-manager:cert-manager"
      }
    },
    "Principal": {
      "Federated": "arn:aws:iam::$AWS_ACCOUNT_ID:oidc-provider/${OIDC_ENDPOINT}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity"
    }
    ]
   }
   EOF

5. Create an IAM role for the cert-manager Operator using the trust policy you created in the previous step:

   $ ROLE_ARN=$(aws iam create-role --role-name "${CLUSTER}-cert-manager-operator" \
      --assume-role-policy-document "file://${SCRATCH}/trust-policy.json" \
      --query Role.Arn --output text)

6. Attach the permissions policy to the role:

   $ aws iam attach-role-policy --role-name "${CLUSTER}-cert-manager-operator" \
     --policy-arn ${POLICY_ARN}

1. Create a project to install the cert-manager Operator into:

   $ oc new-project cert-manager-operator

   Important

   Do not attempt to use more than one cert-manager Operator in your cluster. If you have a community cert-manager Operator installed in your cluster, you must uninstall it before installing the cert-manager Operator for Red Hat OpenShift.

2. Install the cert-manager Operator for Red Hat OpenShift:

   $ cat << EOF | oc apply -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-cert-manager-operator-group
     namespace: cert-manager-operator
   spec:
     targetNamespaces:
     - cert-manager-operator
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: openshift-cert-manager-operator
     namespace: cert-manager-operator
   spec:
     channel: stable-v1
     installPlanApproval: Automatic
     name: openshift-cert-manager-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

   Note

   It takes a few minutes for this Operator to install and complete its set up.

3. Verify that the cert-manager Operator is running:

   $ oc -n cert-manager-operator get pods

   Example output

   NAME                                                        READY   STATUS    RESTARTS   AGE
   cert-manager-operator-controller-manager-84b8799db5-gv8mx   2/2     Running   0          12s

4. Annotate the service account used by the cert-manager pods with the AWS IAM role you created earlier:

   $ oc -n cert-manager annotate serviceaccount cert-manager eks.amazonaws.com/role-arn=${ROLE_ARN}

5. Restart the existing cert-manager controller pod by running the following command:

   $ oc -n cert-manager delete pods -l app.kubernetes.io/name=cert-manager

6. Patch the Operator’s configuration to use external nameservers to prevent DNS-01 challenge resolution issues:

   $ oc patch certmanager.operator.openshift.io/cluster --type merge \
     -p '{"spec":{"controllerConfig":{"overrideArgs":["--dns01-recursive-nameservers-only","--dns01-recursive-nameservers=1.1.1.1:53"]}}}'

7. Create a ClusterIssuer resource to use Let’s Encrypt by running the following command:

   $ cat << EOF | oc apply -f -
   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: letsencrypt-production
   spec:
     acme:
       server: https://acme-v02.api.letsencrypt.org/directory
       email: ${EMAIL}
       # This key doesn't exist, cert-manager creates it
       privateKeySecretRef:
         name: prod-letsencrypt-issuer-account-key
       solvers:
       - dns01:
           route53:
            hostedZoneID: ${ZONE_ID}
            region: ${REGION}
            secretAccessKeySecretRef:
              name: ''
   EOF

8. Verify the ClusterIssuer resource is ready:

   $ oc get clusterissuer.cert-manager.io/letsencrypt-production

   Example output

   NAME                     READY   AGE
   letsencrypt-production   True    47s

1. Create and configure a certificate resource to provision a certificate for the custom domain Ingress Controller:

   Note

   The following example uses a single domain certificate. SAN and wildcard certificates are also supported.

   $ cat << EOF | oc apply -f -
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: custom-domain-ingress-cert
     namespace: openshift-ingress
   spec:
     secretName: custom-domain-ingress-cert-tls
     issuerRef:
        name: letsencrypt-production
        kind: ClusterIssuer
     commonName: "${DOMAIN}"
     dnsNames:
     - "${DOMAIN}"
   EOF

2. Verify the certificate has been issued:

   Note

   It takes a few minutes for this certificate to be issued by Let’s Encrypt. If it takes longer than 5 minutes, run oc -n openshift-ingress describe certificate.cert-manager.io/custom-domain-ingress-cert to see any issues reported by cert-manager.

   $ oc -n openshift-ingress get certificate.cert-manager.io/custom-domain-ingress-cert

   Example output

   NAME                         READY   SECRET                           AGE
   custom-domain-ingress-cert   True    custom-domain-ingress-cert-tls   9m53s

3. Create a new IngressController resource:

   $ cat << EOF | oc apply -f -
   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: custom-domain-ingress
     namespace: openshift-ingress-operator
   spec:
     domain: ${DOMAIN}
     defaultCertificate:
       name: custom-domain-ingress-cert-tls
     endpointPublishingStrategy:
       loadBalancer:
         dnsManagementPolicy: Unmanaged
         providerParameters:
           aws:
             type: NLB
           type: AWS
         scope: External
       type: LoadBalancerService
   EOF

   Warning

   This IngressController example will create an internet accessible Network Load Balancer (NLB) in your AWS account. To provision an internal NLB instead, set the .spec.endpointPublishingStrategy.loadBalancer.scope parameter to Internal before creating the IngressController resource.

4. Verify that your custom domain IngressController has successfully created an external load balancer:

   $ oc -n openshift-ingress get service/router-custom-domain-ingress

   Example output

   NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)                      AGE
   router-custom-domain-ingress   LoadBalancer   172.30.174.34   a309962c3bd6e42c08cadb9202eca683-1f5bbb64a1f1ec65.elb.us-east-1.amazonaws.com   80:31342/TCP,443:31821/TCP   7m28s

5. Prepare a document with the necessary DNS changes to enable DNS resolution for your custom domain Ingress Controller:

   $ INGRESS=$(oc -n openshift-ingress get service/router-custom-domain-ingress -ojsonpath="{.status.loadBalancer.ingress[0].hostname}")
   $ cat << EOF > "${SCRATCH}/create-cname.json"
   {
     "Comment":"Add CNAME to custom domain endpoint",
     "Changes":[{
         "Action":"CREATE",
         "ResourceRecordSet":{
           "Name": "*.${DOMAIN}",
         "Type":"CNAME",
         "TTL":30,
         "ResourceRecords":[{
           "Value": "${INGRESS}"
         }]
       }
     }]
   }
   EOF

6. Submit your changes to Amazon Route 53 for propagation:

   $ aws route53 change-resource-record-sets \
     --hosted-zone-id ${ZONE_ID} \
     --change-batch file://${SCRATCH}/create-cname.json

   Note

   While the wildcard CNAME record avoids the need to create a new record for every new application you deploy using the custom domain Ingress Controller, the certificate that each of these applications use is not a wildcard certificate.

1. Create the necessary OpenShift resources cert-manager requires to manage certificates for OpenShift routes.

   This step creates a new deployment (and therefore a pod) that specifically monitors annotated routes in the cluster. If the issuer-kind and issuer-name annotations are found in a new route, it requests the Issuer (ClusterIssuer in this case) for a new certificate that is unique to this route and which will honor the hostname that was specified while creating the route.

   Note

   If the cluster does not have access to GitHub, you can save the raw contents locally and run oc apply -f localfilename.yaml -n cert-manager.

   $ oc -n cert-manager apply -f https://github.com/cert-manager/openshift-routes/releases/latest/download/cert-manager-openshift-routes.yaml

   The following additional OpenShift resources are also created in this step:

   • ClusterRole - grants permissions to watch and update the routes across the cluster
   • ServiceAccount - uses permissions to run the newly created pod
   • ClusterRoleBinding - binds these two resources

2. Ensure that the new cert-manager-openshift-routes pod is running successfully:

   $ oc -n cert-manager get pods

   Example result

   NAME                                             READY   STATUS    RESTARTS   AGE
   cert-manager-866d8f788c-9kspc                    1/1     Running   0          4h21m
   cert-manager-cainjector-6885c585bd-znws8         1/1     Running   0          4h41m
   cert-manager-openshift-routes-75b6bb44cd-f8kd5   1/1     Running   0          6s
   cert-manager-webhook-8498785dd9-bvfdf            1/1     Running   0          4h41m

1. Create a new project for your sample application:

   $ oc new-project hello-world

2. Deploy a hello world application:

   $ oc -n hello-world new-app --image=docker.io/openshift/hello-openshift

3. Create a route to expose the application from outside the cluster:

   $ oc -n hello-world create route edge --service=hello-openshift hello-openshift-tls --hostname hello.${DOMAIN}

4. Verify the certificate for the route is untrusted:

   $ curl -I https://hello.${DOMAIN}

   Example output

   curl: (60) SSL: no alternative certificate subject name matches target host name 'hello.example.com'
   More details here: https://curl.se/docs/sslcerts.html
   
   curl failed to verify the legitimacy of the server and therefore could not
   establish a secure connection to it. To learn more about this situation and
   how to fix it, please visit the web page mentioned above.

5. Annotate the route to trigger cert-manager to provision a certificate for the custom domain:

   $ oc -n hello-world annotate route hello-openshift-tls cert-manager.io/issuer-kind=ClusterIssuer cert-manager.io/issuer-name=letsencrypt-production

   Note

   It takes 2-3 minutes for the certificate to be created. The renewal of the certificate will automatically be managed by the cert-manager Operator as it approaches expiration.

6. Verify the certificate for the route is now trusted:

   $ curl -I https://hello.${DOMAIN}

   Example output

   HTTP/2 200
   date: Thu, 05 Oct 2023 23:45:33 GMT
   content-length: 17
   content-type: text/plain; charset=utf-8
   set-cookie: 52e4465485b6fb4f8a1b1bed128d0f3b=68676068bb32d24f0f558f094ed8e4d7; path=/; HttpOnly; Secure; SameSite=None
   cache-control: private

1. Before creating the egress IP rules, identify which egress IPs you will use.

   Note

   The egress IPs that you select should exist as a part of the subnets in which the worker nodes are provisioned.

2. Optional: Reserve the egress IPs that you requested to avoid conflicts with the AWS Virtual Private Cloud (VPC) Dynamic Host Configuration Protocol (DHCP) service.

   Request explicit IP reservations on the AWS documentation for CIDR reservations page.

1. Create a new project by running the following command:

   $ oc new-project demo-egress-ns

2. Create the egress rule for all pods within the namespace by running the following command:

   $ cat <<EOF | oc apply -f -
   apiVersion: k8s.ovn.org/v1
   kind: EgressIP
   metadata:
     name: demo-egress-ns
   spec:
     # NOTE: these egress IPs are within the subnet range(s) in which my worker nodes
     #       are deployed.
     egressIPs:
       - 10.10.100.253
       - 10.10.150.253
       - 10.10.200.253
     namespaceSelector:
       matchLabels:
         kubernetes.io/metadata.name: demo-egress-ns
   EOF

1. Create a new project by running the following command:

   $ oc new-project demo-egress-pod

2. Create the egress rule for the pod by running the following command:

   Note

   spec.namespaceSelector is a mandatory field.

   $ cat <<EOF | oc apply -f -
   apiVersion: k8s.ovn.org/v1
   kind: EgressIP
   metadata:
     name: demo-egress-pod
   spec:
     # NOTE: these egress IPs are within the subnet range(s) in which my worker nodes
     #       are deployed.
     egressIPs:
       - 10.10.100.254
       - 10.10.150.254
       - 10.10.200.254
     namespaceSelector:
       matchLabels:
         kubernetes.io/metadata.name: demo-egress-pod
     podSelector:
       matchLabels:
         run: demo-egress-pod
   EOF

1. Clean up your cluster by running the following commands:

   $ oc delete svc demo-service -n default; \
   $ oc delete pod demo-service -n default; \
   $ oc delete project demo-egress-ns; \
   $ oc delete project demo-egress-pod; \
   $ oc delete egressip demo-egress-ns; \
   $ oc delete egressip demo-egress-pod

2. Clean up the assigned node labels by running the following command:

   Warning

   If you rely on node labels for your machine pool, this command replaces those labels. Input your desired labels into the --labels field to ensure your node labels remain.

   $ rosa update machinepool ${ROSA_MACHINE_POOL_NAME} \
     --cluster="${ROSA_CLUSTER_NAME}" \
     --labels ""

1. Log in to your cluster using an account with cluster-admin privileges.

2. Configure an environment variable for your cluster name:

   $ export CLUSTER_NAME=$(oc get infrastructure cluster -o=jsonpath="{.status.infrastructureName}"  | sed 's/-[a-z0-9]\{5\}$//')

3. Ensure all fields output correctly before moving to the next section:

   $ echo "Cluster: ${CLUSTER_NAME}"

   Example output

   Cluster: my-rosa-cluster

1. Verify that you can reach the component routes on their default hostnames.

   You can find the hostnames by querying the lists of routes in the openshift-console and openshift-authentication projects.

   $ oc get routes -n openshift-console
   $ oc get routes -n openshift-authentication

   Example output

   NAME        HOST/PORT                                                                          PATH       SERVICES    PORT    TERMINATION          WILDCARD
   console     console-openshift-console.apps.my-example-cluster-aws.z9a9.p1.openshiftapps.com    ... 1 more  console    https   reencrypt/Redirect   None
   downloads   downloads-openshift-console.apps.my-example-cluster-aws.z9a9.p1.openshiftapps.com  ... 1 more  downloads  http    edge/Redirect        None
   NAME              HOST/PORT                                                             PATH        SERVICES          PORT   TERMINATION            WILDCARD
   oauth-openshift   oauth-openshift.apps.my-example-cluster-aws.z9a9.p1.openshiftapps.com ... 1 more  oauth-openshift   6443   passthrough/Redirect   None

   From this output you can see that our base hostname is z9a9.p1.openshiftapps.com.

2. Get the ID of the default ingress by running the following command:

   $ export INGRESS_ID=$(rosa list ingress -c ${CLUSTER_NAME} -o json | jq -r '.[] | select(.default == true) | .id')

3. Ensure all fields output correctly before moving to the next section:

   $ echo "Ingress ID: ${INGRESS_ID}"

   Example output

   Ingress ID: r3l6

   By running these commands you can see that the default component routes for our cluster are:

   • console-openshift-console.apps.my-example-cluster-aws.z9a9.p1.openshiftapps.com for Console
   • downloads-openshift-console.apps.my-example-cluster-aws.z9a9.p1.openshiftapps.com for Downloads
   • oauth-openshift.apps.my-example-cluster-aws.z9a9.p1.openshiftapps.com for OAuth

1. Generate a certificate for each component route, taking care to set our certificate’s subject (-subj) to the custom domain of the component route we want to use:

   Example

   $ openssl req -newkey rsa:2048 -new -nodes -x509 -days 365 -keyout key-console.pem -out cert-console.pem -subj "/CN=console.my-new-domain.dev"
   $ openssl req -newkey rsa:2048 -new -nodes -x509 -days 365 -keyout key-downloads.pem -out cert-downloads.pem -subj "/CN=downloads.my-new-domain.dev"
   $ openssl req -newkey rsa:2048 -new -nodes -x509 -days 365 -keyout key-oauth.pem -out cert-oauth.pem -subj "/CN=oauth.my-new-domain.dev"

   This generates three pairs of .pem files, key-<component>.pem and cert-<component>.pem.

1. Create three TLS secrets in the openshift-config namespace.

   These become your secret reference when you update the component routes later in this guide.

   $ oc create secret tls console-tls --cert=cert-console.pem --key=key-console.pem -n openshift-config
   $ oc create secret tls downloads-tls --cert=cert-downloads.pem --key=key-downloads.pem -n openshift-config
   $ oc create secret tls oauth-tls --cert=cert-oauth.pem --key=key-oauth.pem -n openshift-config

1. Use the rosa edit ingress command to update your default ingress route with the new base domain and the secret reference associated with it, taking care to update the hostnames for each component route.

   $ rosa edit ingress -c ${CLUSTER_NAME} ${INGRESS_ID} --component-routes 'console: hostname=console.my-new-domain.dev;tlsSecretRef=console-tls,downloads: hostname=downloads.my-new-domain.dev;tlsSecretRef=downloads-tls,oauth: hostname=oauth.my-new-domain.dev;tlsSecretRef=oauth-tls'

   Note

   You can also edit only a subset of the component routes by leaving the component routes you do not want to change set to an empty string. For example, if you only want to change the Console and OAuth server hostnames and TLS certificates, you would run the following command:

   $ rosa edit ingress -c ${CLUSTER_NAME} ${INGRESS_ID} --component-routes 'console: hostname=console.my-new-domain.dev;tlsSecretRef=console-tls,downloads: hostname="";tlsSecretRef="", oauth: hostname=oauth.my-new-domain.dev;tlsSecretRef=oauth-tls'

2. Run the rosa list ingress command to verify that your changes were successfully made:

   $ rosa list ingress -c ${CLUSTER_NAME} -ojson | jq ".[] | select(.id == \"${INGRESS_ID}\") | .component_routes"

   Example output

   {
     "console": {
       "kind": "ComponentRoute",
       "hostname": "console.my-new-domain.dev",
       "tls_secret_ref": "console-tls"
     },
     "downloads": {
       "kind": "ComponentRoute",
       "hostname": "downloads.my-new-domain.dev",
       "tls_secret_ref": "downloads-tls"
     },
     "oauth": {
       "kind": "ComponentRoute",
       "hostname": "oauth.my-new-domain.dev",
       "tls_secret_ref": "oauth-tls"
     }
   }

3. Add your certificate to the truststore on your local system, then confirm that you can access your components at their new routes using your local web browser.

1. Run the following command to create the admin user:

   rosa create admin --cluster=<cluster-name>

   Example output

   W: It is recommended to add an identity provider to login to this cluster. See 'rosa create idp --help' for more information.
   I: Admin account has been added to cluster 'my-rosa-cluster'. It may take up to a minute for the account to become active.
   I: To login, run the following command:
   oc login https://api.my-rosa-cluster.abcd.p1.openshiftapps.com:6443 \
   --username cluster-admin \
   --password FWGYL-2mkJI-00000-00000

2. Copy the log in command returned to you in the previous step and paste it into your terminal. This will log you in to the cluster using the CLI so you can start using the cluster.

   $ oc login https://api.my-rosa-cluster.abcd.p1.openshiftapps.com:6443 \
   >    --username cluster-admin \
   >    --password FWGYL-2mkJI-00000-00000

   Example output

   Login successful.
   
   You have access to 79 projects, the list has been suppressed. You can list all projects with ' projects'
   
   Using project "default".

3. To check that you are logged in as the admin user, run one of the following commands:

   • Option 1:

     $ oc whoami

     Example output

     cluster-admin

   • Option 2:

     oc get all -n openshift-apiserver

     Only an admin user can run this command without errors.

4. You can now use the cluster as an admin user, which will suffice for this tutorial. For actual deployment, it is highly recommended to set up an identity provider, which is explained in the next tutorial.

Procedure

1. To get the GitHub webhook trigger secret, in your terminal, run the following command:

   $ oc get bc/ostoy-microservice -o=jsonpath='{.spec.triggers..github.secret}'

   Example output

   `o_3x9M1qoI2Wj_cz1WiK`

   Important

   You need to use this secret in a later step in this process.

2. To get the GitHub webhook trigger URL from the OSToy’s buildconfig, run the following command:

   $ oc describe bc/ostoy-microservice

   Example output

   [...]
   Webhook GitHub:
   	URL:	https://api.demo1234.openshift.com:443/apis/build.openshift.io/v1/namespaces/ostoy-s2i/buildconfigs/ostoy/webhooks/<secret>/github
   [...]

3. In the GitHub webhook URL, replace the <secret> text with the secret you retrieved. Your URL will resemble the following example output:

   Example output

   https://api.demo1234.openshift.com:443/apis/build.openshift.io/v1/namespaces/ostoy-s2i/buildconfigs/ostoy-microservice/webhooks/o_3x9M1qoI2Wj_czR1WiK/github

4. Setup the webhook URL in GitHub repository.

   1. In your repository, click Settings > Webhooks > Add webhook.

      
   2. Paste the GitHub webhook URL with the Secret included into the "Payload URL" field.
   3. Change the "Content type" to application/json.

   4. Click the Add webhook button.

      

      You should see a message from GitHub stating that your webhook was successfully configured. Now, whenever you push a change to your GitHub repository, a new build automatically starts, and upon a successful build, a new deployment starts.

5. Now, make a change in the source code. Any changes automatically trigger a build and deployment. In this example, the colors that denote the status of your OSToy app are selected randomly. To test the configuration, change the box to only display grayscale.

   1. Go to the source code in your repository https://github.com/<username>/ostoy/blob/master/microservice/app.js.
   2. Edit the file.
   3. Comment out line 8 (containing let randomColor = getRandomColor();).

   4. Uncomment line 9 (containing let randomColor = getRandomGrayScaleColor();).

      7   app.get('/', function(request, response) {
      8   //let randomColor = getRandomColor(); // <-- comment this
      9   let randomColor = getRandomGrayScaleColor(); // <-- uncomment this
      10
      11  response.writeHead(200, {'Content-Type': 'application/json'});

   5. Enter a message for the update, such as "changed box to grayscale colors".
   6. Click Commit at the bottom to commit the changes to the main branch.

6. In your cluster’s web UI, click Builds > Builds to determine the status of the build. After this build is completed, the deployment begins. You can also check the status by running oc status in your terminal.

   

7. After the deployment has finished, return to the OSToy application in your browser. Access the Networking menu item on the left. The box color is now limited to grayscale colors only.