Chapter 10. Installing a cluster into a shared VPC on Google Cloud using Infrastructure Manager templates

Procedure

1. Create a project to host the shared VPC for your OpenShift Container Platform cluster. See Creating and Managing Projects in the Google Cloud documentation.
2. Create a service account in the project that hosts your shared VPC. See Creating a service account in the Google Cloud documentation.

3. Grant the service account the appropriate permissions. You can either grant the individual permissions that follow or assign the Owner role to it. See Granting roles to a service account for specific resources.

   Note

   While making the service account an owner of the project is the easiest way to gain the required permissions, it means that service account has complete control over the project. You must determine if the risk that comes from offering that power is acceptable.

   The service account for the project that hosts the shared VPC network requires the following roles:

   • Compute Network User
   • Compute Security Admin
   • Cloud Infrastructure Manager Admin
   • DNS Administrator
   • Security Admin
   • Network Management Admin

Procedure

1. If you are installing a cluster into a shared VPC, set environment variables for the cluster network and control plane subnet.

   1. Determine the shared VPC network name by running the following command:

      $ gcloud compute networks list

   2. Set the CLUSTER_NETWORK variable by running the following command:

      $ export CLUSTER_NETWORK=$(gcloud compute networks describe <network_name> --format json | jq -r .selfLink)

      <network_name> specifies the name of the network you determined.

   3. List the available network subnets by running the following command:

      $ gcloud compute networks subnets list --network=<network_name>

      <network_name> specifies the name of the network you determined.

   4. Select a subnet from the list, and set the CONTROL_SUBNET variable by running the following command:

      $ export CONTROL_SUBNET=<control_subnet>

      <control_subnet> specifies the name of the subnet you selected from the list of subnets.

2. Copy the template from the Infrastructure Manager template for the internal load balancer section of this topic and save it as 02_lb_int.tf in a directory called 02_lb_int on your computer. This template describes the internal load balancing objects that your cluster requires.

   1. Create an internal load balancer by running the following command:

      $ gcloud infra-manager deployments apply <internal_lb_deployment_name> \
        --location=${REGION} \
        --input-values=infra_id=${INFRA_ID},project=${PROJECT_NAME},region=${REGION},cluster_network=${CLUSTER_NETWORK},control_subnet=${CONTROL_SUBNET},zone_0=${ZONE_0},zone_1=${ZONE_1},zone_2=${ZONE_2} \
        --project=${PROJECT_NAME} \
        --local-source=./02_lb_int \
        --service-account=${INSTALL_SERVICE_ACCOUNT}

      <internal_lb_deployment_name> specifies the name of the internal load balancer deployment you create.

   2. Export the CLUSTER_IP variable by running the following command:

      $ export CLUSTER_IP=$(gcloud compute addresses describe ${INFRA_ID}-cluster-ip --region=${REGION} --format json | jq -r .address)

3. Optional: For a public or externally available cluster, copy the template from the Infrastructure Manager template for the external load balancer section of this topic and save it as 02_lb_ext.tf in a directory called 02_lb_ext on your computer. This template describes the external load balancing objects that your cluster requires.

   1. Create an external load balancer by running the following command:

      $ gcloud infra-manager deployments apply <external_lb_deployment_name> \
        --location=${REGION} \
        --input-values=infra_id=${INFRA_ID},project=${PROJECT_NAME},region=${REGION} \
        --project=${PROJECT_NAME} \
        --local-source=./02_lb_ext \
        --service-account=${INSTALL_SERVICE_ACCOUNT}

      <external_lb_deployment_name> specifies the name of the external load balancer deployment you create.

   2. Export the CLUSTER_PUBLIC_IP variable by running the following command:

      $ export CLUSTER_PUBLIC_IP=$(gcloud compute addresses describe ${INFRA_ID}-cluster-public-ip --region=${REGION} --format json | jq -r .address)

Verification

1. Verify the deployment is active by running the following command:

   $ gcloud infra-manager deployments describe <deployment_name> --format='value(state)'

   Replace <deployment_name> with the name of the deployment you created.

   Example output

   ACTIVE

Procedure

1. Copy the template from the Infrastructure Manager template for the private DNS section of this topic and save it as 02_dns.tf in a folder called 02_dns on your computer. This template describes the private DNS objects that your cluster requires.

2. If you are installing a cluster into a shared VPC, and the host project already has a private DNS zone, skip this step. Create the DNS zone by running the following command:

   $ gcloud infra-manager deployments apply <dns_zone_deployment_name> \
     --location=${REGION} \
     --input-values=infra_id=${INFRA_ID},project=${PROJECT_NAME},region=${REGION},cluster_domain=${CLUSTER_DOMAIN},cluster_network=${CLUSTER_NETWORK} \
     --project=${PROJECT_NAME} \
     --local-source=./02_dns \
     --service-account=${INSTALL_SERVICE_ACCOUNT}

   <dns_zone_deployment_name> specifies the name of the DNS zone deployment you create.

3. The templates do not create DNS entries due to limitations of Infrastructure Manager, so you must create them manually:

   1. Add the internal DNS entries by running the following commands:

      $ if [ -f transaction.yaml ]; then rm transaction.yaml; fi

      $ gcloud dns record-sets transaction start --zone ${INFRA_ID}-private-zone --project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}

      $ gcloud dns record-sets transaction add ${CLUSTER_IP} --name api.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${INFRA_ID}-private-zone --project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}

      $ gcloud dns record-sets transaction add ${CLUSTER_IP} --name api-int.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${INFRA_ID}-private-zone --project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}

      $ gcloud dns record-sets transaction execute --zone ${INFRA_ID}-private-zone --project ${HOST_PROJECT} --account ${HOST_PROJECT_ACCOUNT}

   2. For an external cluster, also add the external DNS entries by running the following commands:

      $ if [ -f transaction.yaml ]; then rm transaction.yaml; fi

      $ gcloud --account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT} dns record-sets transaction start --zone ${BASE_DOMAIN_ZONE_NAME}

      $ gcloud --account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT} dns record-sets transaction add ${CLUSTER_PUBLIC_IP} --name api.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${BASE_DOMAIN_ZONE_NAME}

      $ gcloud --account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT} dns record-sets transaction execute --zone ${BASE_DOMAIN_ZONE_NAME}

Verification

1. Verify the deployment is active by running the following command:

   $ gcloud infra-manager deployments describe <deployment_name> --format='value(state)'

   Replace <deployment_name> with the name of the deployment you created.

   Example output

   ACTIVE

Procedure

1. Copy the template from the Infrastructure Manager template for firewall rules and IAM roles section of this topic and save it as 03_security.tf in a folder called 03_security on your computer. This template describes the security groups that your cluster requires.

2. Create the firewall rules and IAM roles by running the following command:

   $ gcloud infra-manager deployments apply <security_deployment_name> \
     --location=${REGION} \
     --project=${PROJECT_NAME} \
     --local-source=./03_security \
     --input-values=infra_id=${INFRA_ID},project=${PROJECT_NAME},region=${REGION},cluster_network=${CLUSTER_NETWORK},network_cidr=${NETWORK_CIDR} \
     --service-account=${INSTALL_SERVICE_ACCOUNT}

   <security_deployment_name> specifies the name of the deployment of firewall rules and IAM roles.

3. Configure service account variables based on the roles you created by running the following commands:

   $ export MASTER_SERVICE_ACCOUNT=$(gcloud iam service-accounts list --filter "email~^${INFRA_ID}-m@${PROJECT_NAME}." --format json | jq -r '.[0].email')

   $ export WORKER_SERVICE_ACCOUNT=$(gcloud iam service-accounts list --filter "email~^${INFRA_ID}-w@${PROJECT_NAME}." --format json | jq -r '.[0].email')

Verification

1. Verify the deployment is active by running the following command:

   $ gcloud infra-manager deployments describe <deployment_name> --format='value(state)'

   Replace <deployment_name> with the name of the deployment you created.

   Example output

   ACTIVE

Procedure

1. Assign the permissions that the installation program requires to the service accounts for the subnets that host the control plane and compute subnets:

   1. Grant the networkViewer role of the project that hosts your shared VPC to the master service account by running the following command:

      $ gcloud --account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT} projects add-iam-policy-binding ${HOST_PROJECT} --member "serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.networkViewer"

   2. Grant the networkUser role to the master service account for the control plane subnet by running the following command:

      $ gcloud --account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT} compute networks subnets add-iam-policy-binding "${HOST_PROJECT_CONTROL_SUBNET}" --member "serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.networkUser" --region ${REGION}

   3. Grant the networkUser role to the worker service account for the control plane subnet by running the following command:

      $ gcloud --account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT} compute networks subnets add-iam-policy-binding "${HOST_PROJECT_CONTROL_SUBNET}" --member "serviceAccount:${WORKER_SERVICE_ACCOUNT}" --role "roles/compute.networkUser" --region ${REGION}

   4. Grant the networkUser role to the master service account for the compute subnet by running the following command:

      $ gcloud --account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT} compute networks subnets add-iam-policy-binding "${HOST_PROJECT_COMPUTE_SUBNET}" --member "serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.networkUser" --region ${REGION}

   5. Grant the networkUser role to the worker service account for the compute subnet by running the following command:

      $ gcloud --account=${HOST_PROJECT_ACCOUNT} --project=${HOST_PROJECT} compute networks subnets add-iam-policy-binding "${HOST_PROJECT_COMPUTE_SUBNET}" --member "serviceAccount:${WORKER_SERVICE_ACCOUNT}" --role "roles/compute.networkUser" --region ${REGION}

2. The templates do not create the policy bindings due to limitations of Infrastructure Manager, so you must create them manually by running the following commands:

   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.instanceAdmin"

   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.networkAdmin"

   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/compute.securityAdmin"

   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/iam.serviceAccountUser"

   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${MASTER_SERVICE_ACCOUNT}" --role "roles/storage.admin"

   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${WORKER_SERVICE_ACCOUNT}" --role "roles/compute.viewer"

   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${WORKER_SERVICE_ACCOUNT}" --role "roles/storage.admin"

3. Create a service account key and store it locally for later use by running the following command:

   $ gcloud iam service-accounts keys create service-account-key.json --iam-account=${MASTER_SERVICE_ACCOUNT}

Procedure

1. Obtain the image name by running the following command:

   $ source_image=$(openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.images.gcp.name')

2. Obtain the project name by running the following command:

   $ source_project=$(openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.images.gcp.project')

3. Create the image by running the following command:

   $ gcloud compute images create "${INFRA_ID}-rhcos-image" \
       --source-image="${source_image}" --source-image-project="${source_project}"

Procedure

1. Copy the template from the Infrastructure Manager template for the bootstrap machine section of this topic and save it as 04_bootstrap.tf in a folder called 04_bootstrap on your computer. This template describes the bootstrap machine that your cluster requires.

   • You can edit the 04_bootstrap.tf file to add additional tags to the bootstrap machine, by modifying the existing tags stanza as follows:

     resource "google_compute_instance" "bootstrap" {
     # ...
       tags = [
         "${var.infra_id}-master",
         "${var.infra_id}-bootstrap",
         "custom-tag-example"
       ]
     # ...
     }

2. Export the location of the Red Hat Enterprise Linux CoreOS (RHCOS) image that the installation program requires by running the following command:

   $ export CLUSTER_IMAGE=(`gcloud compute images describe ${INFRA_ID}-rhcos-image --format json | jq -r .selfLink`)

3. Create a bucket by running the following command:

   $ gcloud storage buckets create "gs://${INFRA_ID}-bootstrap-ignition"

4. Upload the bootstrap.ign file by running the following command:

   $ gcloud storage cp bootstrap.ign "gs://${INFRA_ID}-bootstrap-ignition/"

5. Create a signed URL for the bootstrap instance and export the URL from the output as a variable by running the following command:

   $ export BOOTSTRAP_IGN="$(gcloud storage sign-url --duration=2h --private-key-file=service-account-key.json "gs://${INFRA_ID}-bootstrap-ignition/bootstrap.ign" | grep "^signed_url:" | awk '{print $2}')"

6. Create the bootstrap deployment by running the following command:

   $ gcloud infra-manager deployments apply <bootstrap_deployment_name> \
     --location=${REGION} \
     --project=${PROJECT_NAME} \
     --local-source=./04_bootstrap \
     --input-values=infra_id=${INFRA_ID},project=${PROJECT_NAME},region=${REGION},zone=${ZONE_0},cluster_network=${CLUSTER_NETWORK},subnet=${CONTROL_SUBNET},image=${CLUSTER_IMAGE},bootstrap_ign="${BOOTSTRAP_IGN}",is_public_cluster=<public_cluster_status> \
     --service-account=${INSTALL_SERVICE_ACCOUNT}

   where:

   <bootstrap_deployment_name>

   Specifies the name of the bootstrap deployment.

   <public_cluster_status>

   Specifies whether the cluster is public or private. If it is a public cluster, specify true. If it is a private cluster, specify false.

7. The templates do not manage load balancer membership due to limitations of Infrastructure Manager, so you must add the bootstrap machine manually.

   1. Add the bootstrap instance to the internal load balancer instance group by running the following command:

      $ gcloud compute instance-groups unmanaged add-instances \
          ${INFRA_ID}-bootstrap-ig --zone=${ZONE_0} --instances=${INFRA_ID}-bootstrap

   2. Add the bootstrap instance group to the internal load balancer backend service by running the following command:

      $ gcloud compute backend-services add-backend \
          ${INFRA_ID}-api-internal --region=${REGION} --instance-group=${INFRA_ID}-bootstrap-ig --instance-group-zone=${ZONE_0}

Verification

1. Verify the deployment is active by running the following command:

   $ gcloud infra-manager deployments describe <deployment_name> --format='value(state)'

   Replace <deployment_name> with the name of the deployment you created.

   Example output

   ACTIVE

Procedure

1. Copy the template from the Infrastructure Manager template for control plane machines section of this topic and save it as 05_control_plane.tf in a folder called 05_control_plane on your computer. This template describes the control plane machines that your cluster requires.

   • You can edit the 05_control_plane.tf file to add additional tags to the control plane machines, by modifying the existing tags stanza. The following example adds a custom tag to the first control plane machine, which is named master_0:

     resource "google_compute_instance" "master_0" {
     # ...
       tags = [
         "${var.infra_id}-master",
         "custom_tag_example"
       ]
     # ...
     }

2. Copy the master.ign file from your installation directory into the 05_control_plane folder by running the following command:

   $ cp <installation_directory>/master.ign 05_control_plane/master.ign

   <installation_directory> specifies the directory where you created the Ignition configuration files.

3. Create the control plane deployment by running the following command:

   $ gcloud infra-manager deployments apply <control_plane_deployment> \
     --location=${REGION} \
     --project=${PROJECT_NAME} \
     --local-source=./05_control_plane \
     --input-values=infra_id=${INFRA_ID},project=${PROJECT_NAME},region=${REGION},zone_0=${ZONE_0},zone_1=${ZONE_1},zone_2=${ZONE_2},subnet=${CONTROL_SUBNET},image=${CLUSTER_IMAGE},service_account_email=${MASTER_SERVICE_ACCOUNT} \
     --service-account=${INSTALL_SERVICE_ACCOUNT}

   <control_plane_deployment> specifies the name of the control plane deployment.

4. Delete the temporary ignition file from the 05_control_plane folder by running the following command:

   $ rm 05_control_plane/master.ign

5. The templates do not manage load balancer membership due to limitations of Infrastructure Manager, so you must add the control plane machines manually.

   1. Add the first control plane machine to an internal load balancer instance group by running the following command:

      $ gcloud compute instance-groups unmanaged add-instances ${INFRA_ID}-master-${ZONE_0}-ig --zone=${ZONE_0} --instances=${INFRA_ID}-master-0

   2. Add the second control plane machine to an internal load balancer instance group by running the following command:

      $ gcloud compute instance-groups unmanaged add-instances ${INFRA_ID}-master-${ZONE_1}-ig --zone=${ZONE_1} --instances=${INFRA_ID}-master-1

   3. Add the third control plane machine to an internal load balancer instance group by running the following command:

      $ gcloud compute instance-groups unmanaged add-instances ${INFRA_ID}-master-${ZONE_2}-ig --zone=${ZONE_2} --instances=${INFRA_ID}-master-2

6. For an external cluster, you must also add the control plane machines to external load balancer target pools.

   1. Add the first control plane machine to an external load balancer pool by running the following command:

      $ gcloud compute target-pools add-instances ${INFRA_ID}-api-target-pool --instances-zone="${ZONE_0}" --instances=${INFRA_ID}-master-0

   2. Add the second control plane machine to an external load balancer pool by running the following command:

      $ gcloud compute target-pools add-instances ${INFRA_ID}-api-target-pool --instances-zone="${ZONE_1}" --instances=${INFRA_ID}-master-1

   3. Add the third control plane machine to an external load balancer pool by running the following command:

      $ gcloud compute target-pools add-instances ${INFRA_ID}-api-target-pool --instances-zone="${ZONE_2}" --instances=${INFRA_ID}-master-2

Verification

1. Verify the deployment is active by running the following command:

   $ gcloud infra-manager deployments describe <deployment_name> --format='value(state)'

   Replace <deployment_name> with the name of the deployment you created.

   Example output

   ACTIVE

Procedure

1. Copy the template from the Infrastructure Manager template for worker machines section of this topic and save it as 06_worker.tf in a folder called 06_worker on your computer. This template describes the worker machines that your cluster requires.

   • You can edit the 06_worker.tf file to add additional tags to the compute machines, by modifying the existing tags stanza as follows:

     resource "google_compute_instance" "worker_0" {
     # ...
       tags = [
         "${var.infra_id}-worker-0",
         "custom-tag-example"
       ]
     # ...
     }

2. Copy the worker.ign file from your installation directory into the 06_worker folder by running the following command:

   $ cp <installation_directory>/worker.ign 06_worker/worker.ign

   <installation_directory> specifies the directory where you created the Ignition configuration files.

3. Create the deployment by running the following command:

   $ gcloud infra-manager deployments apply <worker_deployment_name> \
     --location=${REGION} \
     --project=${PROJECT_NAME} \
     --local-source=./06_worker \
     --input-values=infra_id=${INFRA_ID},project=${PROJECT_NAME},region=${REGION},zone_0=${ZONE_0},zone_1=${ZONE_1},subnet=${COMPUTE_SUBNET},image=${CLUSTER_IMAGE},service_account_email=${WORKER_SERVICE_ACCOUNT} \
     --service-account=${INSTALL_SERVICE_ACCOUNT}

   <worker_deployment_name> specifies the name of the deployment.

4. Remove the worker.ign file by running the following command:

   $ rm 06_worker/worker.ign

Verification

1. Verify the deployment is active by running the following command:

   $ gcloud infra-manager deployments describe <deployment_name> --format='value(state)'

   Replace <deployment_name> with the name of the deployment you created.

   Example output

   ACTIVE

Procedure

1. Change to the directory that includes the installation program and run the following command:

   $ ./openshift-install wait-for bootstrap-complete --dir <installation_directory> \ 

   1

   
       --log-level info 

   2

   1

   For <installation_directory>, specify the path to the directory where you stored the installation files.

   2

   To view different installation details, specify warn, debug, or error instead of info.

   If the command exits without a FATAL warning, your production control plane has initialized.

2. To remove the bootstrap instance group from the backend services' backends, run the following commands:

   $ gcloud compute backend-services remove-backend ${INFRA_ID}-api-internal --region=${REGION} --instance-group=${INFRA_ID}-bootstrap-ig --instance-group-zone=${ZONE_0}

   $ ingress_backendservice=$(gcloud compute backend-services list --filter="backends.group~${INFRA_ID}" --format='value(name)' | grep -v "${INFRA_ID}")

   1. If ingress_backendservice is not empty, run the following describe command for the bootstrap group:

      $ gcloud compute backend-services describe ${ingress_backendservice} --region=${REGION}

   2. If the describe command displays that the bootstrap group is one of its backends, run the following remove-backend command to remove the bootstrap group from the backends:

      $ gcloud compute backend-services remove-backend ${ingress_backendservice} --region=${REGION} --instance-group=${INFRA_ID}-bootstrap-ig --instance-group-zone=${ZONE_0}

   3. To remove the bucket and the deployment, run the following commands:

      $ gcloud storage rm "gs://${INFRA_ID}-bootstrap-ignition/bootstrap.ign"

      $ gcloud storage rm --recursive "gs://${INFRA_ID}-bootstrap-ignition/"

      $ gcloud infra-manager deployments delete <bootstrap_deployment_name> \
          --project=${PROJECT_NAME} --location=${REGION} --quiet

      Specify the name of the bootstrap deployment you created for <bootstrap_deployment_name>.

Procedure

1. Navigate to the Download OpenShift Container Platform page on the Red Hat Customer Portal.
2. Select the architecture from the Product Variant list.
3. Select the appropriate version from the Version list.
4. Click Download Now next to the OpenShift v4.22 Linux Clients entry and save the file.

5. Unpack the archive:

   $ tar xvf <file>

6. Place the oc binary in a directory that is on your PATH.

   To check your PATH, execute the following command:

   $ echo $PATH

Procedure

1. Navigate to the Download OpenShift Container Platform page on the Red Hat Customer Portal.
2. Select the appropriate version from the Version list.
3. Click Download Now next to the OpenShift v4.22 Windows Client entry and save the file.
4. Extract the archive with a ZIP program.

5. Move the oc binary to a directory that is on your PATH variable.

   To check your PATH variable, open the command prompt and execute the following command:

   C:\> path

Procedure

1. Navigate to the Download OpenShift Container Platform page on the Red Hat Customer Portal.
2. Select the architecture from the Product Variant list.
3. Select the appropriate version from the Version list.

4. Click Download Now next to the OpenShift v4.22 macOS Clients entry and save the file.

   Note

   For macOS arm64, choose the OpenShift v4.22 macOS arm64 Client entry.

5. Unpack and unzip the archive.

6. Move the oc binary to a directory on your PATH variable.

   To check your PATH variable, open a terminal and execute the following command:

   $ echo $PATH

Procedure

1. Export the kubeadmin credentials by running the following command:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig

   where:

   <installation_directory>

   Specifies the path to the directory that stores the installation files.

2. Verify you can run oc commands successfully using the exported configuration by running the following command:

   $ oc whoami

   Example output

   system:admin

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.35.4
   master-1  Ready     master  63m  v1.35.4
   master-2  Ready     master  64m  v1.35.4

   The output lists all of the machines that you created.

   Note

   The preceding output might not include the compute nodes, also known as worker nodes, until some CSRs are approved.

2. Review the pending CSRs and ensure that you see the client requests with the Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   ...

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   You must approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After the client CSR is approved, the Kubelet creates a secondary CSR for the serving certificate, which requires manual approval. Then, subsequent serving certificate renewal requests are automatically approved by the machine-approver if the Kubelet requests a new certificate with identical parameters.

   Note

   For clusters running on platforms that are not machine API enabled, such as bare metal and other user-provisioned infrastructure, you must implement a method of automatically approving the kubelet serving certificate requests (CSRs). If a request is not approved, then the oc exec, oc rsh, and oc logs commands cannot succeed, because a serving certificate is required when the API server connects to the kubelet. Any operation that contacts the Kubelet endpoint requires this certificate approval to be in place. The method must watch for new CSRs, confirm that the CSR was submitted by the node-bootstrapper service account in the system:node or system:admin groups, and confirm the identity of the node.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name>

     where:

     <csr_name>

     Specifies the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

     Note

     Some Operators might not become available until some CSRs are approved. Each node submits two CSRs, so you may need to run the command to approve CSRs multiple times.

4. Now that your client requests are approved, you must review the server requests for each machine that you added to the cluster:

   $ oc get csr

   Example output

   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

5. If the remaining CSRs are not approved, and are in the Pending status, approve the CSRs for your cluster machines:

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name>

     where:

     <csr_name>

     Specifies the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

6. After all client and server CSRs have been approved, the machines have the Ready status. Verify this by running the following command:

   $ oc get nodes

   Example output

   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  73m  v1.35.4
   master-1  Ready     master  73m  v1.35.4
   master-2  Ready     master  74m  v1.35.4
   worker-0  Ready     worker  11m  v1.35.4
   worker-1  Ready     worker  11m  v1.35.4

   Note

   It can take a few minutes after approval of the server CSRs for the machines to transition to the Ready status.

Procedure

1. Wait for the Ingress router to create a load balancer and populate the EXTERNAL-IP field:

   $ oc -n openshift-ingress get service router-default

   Example output

   NAME             TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
   router-default   LoadBalancer   172.30.18.154   35.233.157.184   80:32288/TCP,443:31215/TCP   98

2. Add the A record to your zones:

   • To use A records:

     1. Export the variable for the router IP address:

        $ export ROUTER_IP=`oc -n openshift-ingress get service router-default --no-headers | awk '{print $4}'`

     2. Add the A record to the private zones:

        +

1. For an external cluster, also add the A record to the public zones:

   +

Procedure

1. Complete the cluster installation:

   $ ./openshift-install --dir <installation_directory> wait-for install-complete 

   1

   Example output

   INFO Waiting up to 30m0s for the cluster to initialize...

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

   Important
   • The Ignition config files that the installation program generates contain certificates that expire after 24 hours, which are then renewed at that time. If the cluster is shut down before renewing the certificates and the cluster is later restarted after the 24 hours have elapsed, the cluster automatically recovers the expired certificates. The exception is that you must manually approve the pending node-bootstrapper certificate signing requests (CSRs) to recover kubelet certificates. See the documentation for Recovering from expired control plane certificates for more information.
   • It is recommended that you use Ignition config files within 12 hours after they are generated because the 24-hour certificate rotates from 16 to 22 hours after the cluster is installed. By using the Ignition config files within 12 hours, you can avoid installation failure if the certificate update runs during installation.

2. Observe the running state of your cluster.

   1. Run the following command to view the current cluster version and status:

      $ oc get clusterversion

      Example output

      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version             False       True          24m     Working towards 4.5.4: 99% complete

   2. Run the following command to view the Operators managed on the control plane by the Cluster Version Operator (CVO):

      $ oc get clusteroperators

      Example output

      NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
      authentication                             4.5.4     True        False         False      7m56s
      cloud-credential                           4.5.4     True        False         False      31m
      cluster-autoscaler                         4.5.4     True        False         False      16m
      console                                    4.5.4     True        False         False      10m
      csi-snapshot-controller                    4.5.4     True        False         False      16m
      dns                                        4.5.4     True        False         False      22m
      etcd                                       4.5.4     False       False         False      25s
      image-registry                             4.5.4     True        False         False      16m
      ingress                                    4.5.4     True        False         False      16m
      insights                                   4.5.4     True        False         False      17m
      kube-apiserver                             4.5.4     True        False         False      19m
      kube-controller-manager                    4.5.4     True        False         False      20m
      kube-scheduler                             4.5.4     True        False         False      20m
      kube-storage-version-migrator              4.5.4     True        False         False      16m
      machine-api                                4.5.4     True        False         False      22m
      machine-config                             4.5.4     True        False         False      22m
      marketplace                                4.5.4     True        False         False      16m
      monitoring                                 4.5.4     True        False         False      10m
      network                                    4.5.4     True        False         False      23m
      node-tuning                                4.5.4     True        False         False      23m
      openshift-apiserver                        4.5.4     True        False         False      17m
      openshift-controller-manager               4.5.4     True        False         False      15m
      openshift-samples                          4.5.4     True        False         False      16m
      operator-lifecycle-manager                 4.5.4     True        False         False      22m
      operator-lifecycle-manager-catalog         4.5.4     True        False         False      22m
      operator-lifecycle-manager-packageserver   4.5.4     True        False         False      18m
      service-ca                                 4.5.4     True        False         False      23m
      service-catalog-apiserver                  4.5.4     True        False         False      23m
      service-catalog-controller-manager         4.5.4     True        False         False      23m
      storage                                    4.5.4     True        False         False      17m

   3. Run the following command to view your cluster pods:

      $ oc get pods --all-namespaces

      Example output

      NAMESPACE                                               NAME                                                                READY     STATUS      RESTARTS   AGE
      kube-system                                             etcd-member-ip-10-0-3-111.us-east-2.compute.internal                1/1       Running     0          35m
      kube-system                                             etcd-member-ip-10-0-3-239.us-east-2.compute.internal                1/1       Running     0          37m
      kube-system                                             etcd-member-ip-10-0-3-24.us-east-2.compute.internal                 1/1       Running     0          35m
      openshift-apiserver-operator                            openshift-apiserver-operator-6d6674f4f4-h7t2t                       1/1       Running     1          37m
      openshift-apiserver                                     apiserver-fm48r                                                     1/1       Running     0          30m
      openshift-apiserver                                     apiserver-fxkvv                                                     1/1       Running     0          29m
      openshift-apiserver                                     apiserver-q85nm                                                     1/1       Running     0          29m
      ...
      openshift-service-ca-operator                           openshift-service-ca-operator-66ff6dc6cd-9r257                      1/1       Running     0          37m
      openshift-service-ca                                    apiservice-cabundle-injector-695b6bcbc-cl5hm                        1/1       Running     0          35m
      openshift-service-ca                                    configmap-cabundle-injector-8498544d7-25qn6                         1/1       Running     0          35m
      openshift-service-ca                                    service-serving-cert-signer-6445fc9c6-wqdqn                         1/1       Running     0          35m
      openshift-service-catalog-apiserver-operator            openshift-service-catalog-apiserver-operator-549f44668b-b5q2w       1/1       Running     0          32m
      openshift-service-catalog-controller-manager-operator   openshift-service-catalog-controller-manager-operator-b78cr2lnm     1/1       Running     0          31m

      When the current cluster version is AVAILABLE, the installation is complete.