Chapter 1. Installing on GCP

Procedure

1. Identify your domain, or subdomain, and registrar. You can transfer an existing domain and registrar or obtain a new one through GCP or another source.

   Note

   If you purchase a new domain, it can take time for the relevant DNS changes to propagate. For more information about purchasing domains through Google, see Google Domains.

2. Create a public hosted zone for your domain or subdomain in your GCP project. See Creating public zones in the GCP documentation.

   Use an appropriate root domain, such as openshiftcorp.com, or subdomain, such as clusters.openshiftcorp.com.

3. Extract the new authoritative name servers from the hosted zone records. See Look up your Cloud DNS name servers in the GCP documentation.

   You typically have four name servers.

4. Update the registrar records for the name servers that your domain uses. For example, if you registered your domain to Google Domains, see the following topic in the Google Domains Help: How to switch to custom name servers.
5. If you migrated your root domain to Google Cloud DNS, migrate your DNS records. See Migrating to Cloud DNS in the GCP documentation.
6. If you use a subdomain, follow your company’s procedures to add its delegation records to the parent domain. This process might include a request to your company’s IT department or the division that controls the root domain and DNS services for your company.

Procedure

1. Create a service account in the project that you use to host your OpenShift Container Platform cluster. See Creating a service account in the GCP documentation.

2. Grant the service account the appropriate permissions. You can either grant the individual permissions that follow or assign the Owner role to it. See Granting roles to a service account for specific resources.

   Note

   While making the service account an Owner of the project is the easiest way to gain the required permissions, it means that that service account has complete control over the project. You must determine if the risk that comes from offering that power is acceptable.

3. Create the service account key in JSON format. See Creating service account keys in the GCP documentation.

   The service account key is required to create a cluster.

Procedure

1. If you do not have an SSH key that is configured for password-less authentication on your computer, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t rsa -b 4096 -N '' \
       -f <path>/<file_name> 1

   1

   Specify the path and file name, such as ~/.ssh/id_rsa, of the SSH key.

   Running this command generates an SSH key that does not require a password in the location that you specified.

2. Start the ssh-agent process as a background task:

   $ eval "$(ssh-agent -s)"
   
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> 1
   
   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   1

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_rsa

Procedure

1. Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

2. Navigate to the page for your installation type, download the installation program for your operating system, and place the file in the directory where you will store the installation configuration files.

   Important

   The installation program creates several files on the computer that you use to install your cluster. You must keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar xvf <installation_program>.tar.gz

4. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret as a .txt file. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

Procedure

1. Remove any existing GCP credentials that do not use the service account key for the GCP account that you configured for your cluster and that are stored in the following locations:

   • The GOOGLE_CREDENTIALS, GOOGLE_CLOUD_KEYFILE_JSON, or GCLOUD_KEYFILE_JSON environment variables
   • The ~/.gcp/osServiceAccount.json file
   • The gcloud cli default credentials

2. Run the installation program:

   $ ./openshift-install create cluster --dir=<installation_directory> \ 1
       --log-level=info 2

   1

   For <installation_directory>, specify the directory name to store the files that the installation program creates.

   2

   To view different installation details, specify warn, debug, or error instead of info.

   Important

   Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

   Provide values at the prompts:

   1. Optional: Select an SSH key to use to access your cluster machines.

      Note

      For production OpenShift Container Platform clusters on which you want to perform installation debugging or disaster recovery on, specify an SSH key that your ssh-agent process uses.

   2. Select gcp as the platform to target.
   3. If you have not configured the service account key for your GCP account on your computer, you must obtain it from GCP and paste the contents of the file or enter the absolute path to the file.
   4. Select the project ID to provision the cluster in. The default value is specified by the service account that you configured.
   5. Select the region to deploy the cluster to.
   6. Select the base domain to deploy the cluster to. The base domain corresponds to the public DNS zone that you created for your cluster.
   7. Enter a descriptive name for your cluster. If you provide a name that is longer than 6 characters, only the first 6 characters will be used in the infrastructure ID that is generated from the cluster name.
   8. Paste the pull secret that you obtained from the Pull Secret page on the Red Hat OpenShift Cluster Manager site.
   Note

   If the cloud provider account that you configured on your host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed.

   When the cluster deployment completes, directions for accessing your cluster, including a link to its web console and credentials for the kubeadmin user, display in your terminal.

   Important

   The Ignition config files that the installation program generates contain certificates that expire after 24 hours. You must keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

   Important

   You must not delete the installation program or the files that the installation program creates. Both are required to delete the cluster.

3. Optional: You can reduce the number of permissions for the service account that you used to install the cluster.

   • If you assigned the Owner role to your service account, you can remove that role and replace it with the Viewer role.
   • If you included the Service Account Key Admin role, you can remove it.

Procedure

1. From the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site, navigate to the page for your installation type and click Download Command-line Tools.

2. Click the folder for your operating system and architecture and click the compressed file.

   Note

   You can install oc on Linux, Windows, or macOS.

3. Save the file to your file system.
4. Extract the compressed file.
5. Place it in a directory that is on your PATH.

Procedure

1. Export the kubeadmin credentials:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig 1

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

2. Verify you can run oc commands successfully using the exported configuration:

   $ oc whoami
   system:admin

Procedure

1. If you do not have an SSH key that is configured for password-less authentication on your computer, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t rsa -b 4096 -N '' \
       -f <path>/<file_name> 1

   1

   Specify the path and file name, such as ~/.ssh/id_rsa, of the SSH key.

   Running this command generates an SSH key that does not require a password in the location that you specified.

2. Start the ssh-agent process as a background task:

   $ eval "$(ssh-agent -s)"
   
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> 1
   
   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   1

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_rsa

Procedure

1. Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

2. Navigate to the page for your installation type, download the installation program for your operating system, and place the file in the directory where you will store the installation configuration files.

   Important

   The installation program creates several files on the computer that you use to install your cluster. You must keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar xvf <installation_program>.tar.gz

4. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret as a .txt file. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

Procedure

1. Create the install-config.yaml file.

   1. Run the following command:

      $ ./openshift-install create install-config --dir=<installation_directory> 1

      1

      For <installation_directory>, specify the directory name to store the files that the installation program creates.

      Important

      Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

   2. At the prompts, provide the configuration details for your cloud:

      1. Optional: Select an SSH key to use to access your cluster machines.

         Note

         For production OpenShift Container Platform clusters on which you want to perform installation debugging or disaster recovery on, specify an SSH key that your ssh-agent process uses.

      2. Select gcp as the platform to target.
      3. If you have not configured the service account key for your GCP account on your computer, you must obtain it from GCP and paste the contents of the file or enter the absolute path to the file.
      4. Select the project ID to provision the cluster in. The default value is specified by the service account that you configured.
      5. Select the region to deploy the cluster to.
      6. Select the base domain to deploy the cluster to. The base domain corresponds to the public DNS zone that you created for your cluster.
      7. Enter a descriptive name for your cluster. If you provide a name that is longer than 6 characters, only the first 6 characters will be used in the infrastructure ID that is generated from the cluster name.
      8. Paste the pull secret that you obtained from the Pull Secret page on the Red Hat OpenShift Cluster Manager site.
2. Modify the install-config.yaml file. You can find more information about the available parameters in the Installation configuration parameters section.

3. Back up the install-config.yaml file so that you can use it to install multiple clusters.

   Important

   The install-config.yaml file is consumed during the installation process. If you want to reuse the file, you must back it up now.

Procedure

1. Remove any existing GCP credentials that do not use the service account key for the GCP account that you configured for your cluster and that are stored in the following locations:

   • The GOOGLE_CREDENTIALS, GOOGLE_CLOUD_KEYFILE_JSON, or GCLOUD_KEYFILE_JSON environment variables
   • The ~/.gcp/osServiceAccount.json file
   • The gcloud cli default credentials

2. Run the installation program:

   $ ./openshift-install create cluster --dir=<installation_directory> \ 1
       --log-level=info 2

   1

   For <installation_directory>, specify the location of your customized ./install-config.yaml file.

   2

   To view different installation details, specify warn, debug, or error instead of info.

   Note

   If the cloud provider account that you configured on your host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed.

   When the cluster deployment completes, directions for accessing your cluster, including a link to its web console and credentials for the kubeadmin user, display in your terminal.

   Important

   The Ignition config files that the installation program generates contain certificates that expire after 24 hours. You must keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

   Important

   You must not delete the installation program or the files that the installation program creates. Both are required to delete the cluster.

3. Optional: You can reduce the number of permissions for the service account that you used to install the cluster.

   • If you assigned the Owner role to your service account, you can remove that role and replace it with the Viewer role.
   • If you included the Service Account Key Admin role, you can remove it.

Procedure

1. From the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site, navigate to the page for your installation type and click Download Command-line Tools.

2. Click the folder for your operating system and architecture and click the compressed file.

   Note

   You can install oc on Linux, Windows, or macOS.

3. Save the file to your file system.
4. Extract the compressed file.
5. Place it in a directory that is on your PATH.

Procedure

1. Export the kubeadmin credentials:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig 1

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

2. Verify you can run oc commands successfully using the exported configuration:

   $ oc whoami
   system:admin

Procedure

1. If you do not have an SSH key that is configured for password-less authentication on your computer, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t rsa -b 4096 -N '' \
       -f <path>/<file_name> 1

   1

   Specify the path and file name, such as ~/.ssh/id_rsa, of the SSH key.

   Running this command generates an SSH key that does not require a password in the location that you specified.

2. Start the ssh-agent process as a background task:

   $ eval "$(ssh-agent -s)"
   
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> 1
   
   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   1

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_rsa

Procedure

1. Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

2. Navigate to the page for your installation type, download the installation program for your operating system, and place the file in the directory where you will store the installation configuration files.

   Important

   The installation program creates several files on the computer that you use to install your cluster. You must keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar xvf <installation_program>.tar.gz

4. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret as a .txt file. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

Procedure

1. Create the install-config.yaml file.

   1. Run the following command:

      $ ./openshift-install create install-config --dir=<installation_directory> 1

      1

      For <installation_directory>, specify the directory name to store the files that the installation program creates.

      Important

      Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

   2. At the prompts, provide the configuration details for your cloud:

      1. Optional: Select an SSH key to use to access your cluster machines.

         Note

         For production OpenShift Container Platform clusters on which you want to perform installation debugging or disaster recovery on, specify an SSH key that your ssh-agent process uses.

      2. Select gcp as the platform to target.
      3. If you have not configured the service account key for your GCP account on your computer, you must obtain it from GCP and paste the contents of the file or enter the absolute path to the file.
      4. Select the project ID to provision the cluster in. The default value is specified by the service account that you configured.
      5. Select the region to deploy the cluster to.
      6. Select the base domain to deploy the cluster to. The base domain corresponds to the public DNS zone that you created for your cluster.
      7. Enter a descriptive name for your cluster. If you provide a name that is longer than 6 characters, only the first 6 characters will be used in the infrastructure ID that is generated from the cluster name.
      8. Paste the pull secret that you obtained from the Pull Secret page on the Red Hat OpenShift Cluster Manager site.
2. Modify the install-config.yaml file. You can find more information about the available parameters in the Installation configuration parameters section.

3. Back up the install-config.yaml file so that you can use it to install multiple clusters.

   Important

   The install-config.yaml file is consumed during the installation process. If you want to reuse the file, you must back it up now.

Procedure

1. Use the following command to create manifests:

   $ ./openshift-install create manifests --dir=<installation_directory> 1

   1

   For <installation_directory>, specify the name of the directory that contains the install-config.yaml file for your cluster.

2. Modify the <installation_directory>/manifests/cluster-scheduler-02-config.yml Kubernetes manifest file to prevent Pods from being scheduled on the control plane machines:

   1. Open the manifests/cluster-scheduler-02-config.yml file.
   2. Locate the mastersSchedulable parameter and set its value to False.
   3. Save and exit the file.
   Note

   Currently, due to a Kubernetes limitation, router Pods running on control plane machines will not be reachable by the ingress load balancer.

3. Create a file that is named cluster-network-03-config.yml in the <installation_directory>/manifests/ directory:

   $ touch <installation_directory>/manifests/cluster-network-03-config.yml 1

   1

   For <installation_directory>, specify the directory name that contains the manifests/ directory for your cluster.

   After creating the file, several network configuration files are in the manifests/ directory, as shown:

   $ ls <installation_directory>/manifests/cluster-network-*
   cluster-network-01-crd.yml
   cluster-network-02-config.yml
   cluster-network-03-config.yml

4. Open the cluster-network-03-config.yml file in an editor and enter a CR that describes the Operator configuration you want:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec: 1
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     serviceNetwork:
     - 172.30.0.0/16
     defaultNetwork:
       type: OpenShiftSDN
       openshiftSDNConfig:
         mode: NetworkPolicy
         mtu: 1450
         vxlanPort: 4789

   1

   The parameters for the spec parameter are only an example. Specify your configuration for the Cluster Network Operator in the CR.

   The CNO provides default values for the parameters in the CR, so you must specify only the parameters that you want to change.

5. Save the cluster-network-03-config.yml file and quit the text editor.
6. Optional: Back up the manifests/cluster-network-03-config.yml file. The installation program deletes the manifests/ directory when creating the cluster.

Procedure

1. Run the installation program:

   $ ./openshift-install create cluster --dir=<installation_directory> \ 1
       --log-level=info 2

   1

   For <installation_directory>, specify the

   2

   To view different installation details, specify warn, debug, or error instead of info.

   Note

   If the cloud provider account that you configured on your host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed.

   When the cluster deployment completes, directions for accessing your cluster, including a link to its web console and credentials for the kubeadmin user, display in your terminal.

   Important

   The Ignition config files that the installation program generates contain certificates that expire after 24 hours. You must keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

   Important

   You must not delete the installation program or the files that the installation program creates. Both are required to delete the cluster.

Procedure

1. From the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site, navigate to the page for your installation type and click Download Command-line Tools.

2. Click the folder for your operating system and architecture and click the compressed file.

   Note

   You can install oc on Linux, Windows, or macOS.

3. Save the file to your file system.
4. Extract the compressed file.
5. Place it in a directory that is on your PATH.

Procedure

1. Export the kubeadmin credentials:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig 1

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

2. Verify you can run oc commands successfully using the exported configuration:

   $ oc whoami
   system:admin

Procedure

1. Copy the template from the Deployment Manager template for the VPC section of this topic and save it as 01_vpc.py on your computer. This template describes the VPC that your cluster requires.

2. Create a 01_vpc.yaml resource definition file:

   $ cat <<EOF >01_vpc.yaml
   imports:
   - path: 01_vpc.py
   
   resources:
   - name: cluster-vpc
    type: 01_vpc.py
    properties:
      infra_id: '${INFRA_ID}' 1
      region: '${REGION}' 2
   
      master_subnet_cidr: '${MASTER_SUBNET_CIDR}' 3
      worker_subnet_cidr: '${WORKER_SUBNET_CIDR}' 4
   EOF

   1

   infra_id is the INFRA_ID infrastructure name from the extraction step.

   2

   region is the region to deploy the cluster into, for example us-east1.

   3

   master_subnet_cidr is the CIDR for the master subnet, for example 10.0.0.0/19.

   4

   worker_subnet_cidr is the CIDR for the worker subnet, for example 10.0.32.0/19.

3. Create the deployment by using the gcloud CLI:

   $ gcloud deployment-manager deployments create ${INFRA_ID}-vpc --config 01_vpc.yaml

Procedure

1. Copy the template from the Deployment Manager template for the network and load balancers section of this topic and save it as 02_infra.py on your computer. This template describes the networking and load balancing objects that your cluster requires.

2. Export the following variable required by the resource definition:

   $ export CLUSTER_NETWORK=`gcloud compute networks describe ${INFRA_ID}-network --format json | jq -r .selfLink`

3. Create a 02_infra.yaml resource definition file:

   $ cat <<EOF >02_infra.yaml
   imports:
   - path: 02_infra.py
   
   resources:
   - name: cluster-infra
     type: 02_infra.py
     properties:
       infra_id: '${INFRA_ID}' 1
       region: '${REGION}' 2
   
       cluster_domain: '${CLUSTER_NAME}.${BASE_DOMAIN}' 3
       cluster_network: '${CLUSTER_NETWORK}' 4
   EOF

   1

   infra_id is the INFRA_ID infrastructure name from the extraction step.

   2

   region is the region to deploy the cluster into, for example us-east1.

   3

   cluster_domain is the domain for the cluster, for example openshift.example.com.

   4

   cluster_network is the selfLink URL to the cluster network.

4. Create the deployment by using the gcloud CLI:

   $ gcloud deployment-manager deployments create ${INFRA_ID}-infra --config 02_infra.yaml

5. The templates do not create DNS entries due to limitations of Deployment Manager, so you must create them manually:

   1. Export the following variable:

      $ export CLUSTER_IP=`gcloud compute addresses describe ${INFRA_ID}-cluster-public-ip --region=${REGION} --format json | jq -r .address`

   2. Add external DNS entries:

      $ if [ -f transaction.yaml ]; then rm transaction.yaml; fi
      $ gcloud dns record-sets transaction start --zone ${BASE_DOMAIN_ZONE_NAME}
      $ gcloud dns record-sets transaction add ${CLUSTER_IP} --name api.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${BASE_DOMAIN_ZONE_NAME}
      $ gcloud dns record-sets transaction execute --zone ${BASE_DOMAIN_ZONE_NAME}

   3. Add internal DNS entries:

      $ if [ -f transaction.yaml ]; then rm transaction.yaml; fi
      $ gcloud dns record-sets transaction start --zone ${INFRA_ID}-private-zone
      $ gcloud dns record-sets transaction add ${CLUSTER_IP} --name api.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${INFRA_ID}-private-zone
      $ gcloud dns record-sets transaction add ${CLUSTER_IP} --name api-int.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${INFRA_ID}-private-zone
      $ gcloud dns record-sets transaction execute --zone ${INFRA_ID}-private-zone

Procedure

1. Copy the template from the Deployment Manager template for firewall rules and IAM roles section of this topic and save it as 03_security.py on your computer. This template describes the security groups and roles that your cluster requires.

2. Export the following variables required by the resource definition:

   $ export MASTER_NAT_IP=`gcloud compute addresses describe ${INFRA_ID}-master-nat-ip --region ${REGION} --format json | jq -r .address`
   $ export WORKER_NAT_IP=`gcloud compute addresses describe ${INFRA_ID}-worker-nat-ip --region ${REGION} --format json | jq -r .address`

3. Create a 03_security.yaml resource definition file:

   $ cat <<EOF >03_security.yaml
   imports:
   - path: 03_security.py
   
   resources:
   - name: cluster-security
     type: 03_security.py
     properties:
       infra_id: '${INFRA_ID}' 1
       region: '${REGION}' 2
   
       cluster_network: '${CLUSTER_NETWORK}' 3
       network_cidr: '${NETWORK_CIDR}' 4
       master_nat_ip: '${MASTER_NAT_IP}' 5
       worker_nat_ip: '${WORKER_NAT_IP}' 6
   EOF

   1

   infra_id is the INFRA_ID infrastructure name from the extraction step.

   2

   region is the region to deploy the cluster into, for example us-east1.

   3

   cluster_network is the selfLink URL to the cluster network.

   4

   network_cidr is the CIDR of the VPC network, for example 10.0.0.0/16.

   5

   master_nat_ip is the IP address of the master NAT, for example 34.94.100.1.

   6

   worker_nat_ip is the IP address of the worker NAT, for example 34.94.200.1.

4. Create the deployment by using the gcloud CLI:

   $ gcloud deployment-manager deployments create ${INFRA_ID}-security --config 03_security.yaml

5. The templates do not create the policy bindings due to limitations of Deployment Manager, so you must create them manually:

   $ export MASTER_SA=${INFRA_ID}-m@${PROJECT_NAME}.iam.gserviceaccount.com
   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${MASTER_SA}" --role "roles/compute.instanceAdmin"
   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${MASTER_SA}" --role "roles/compute.networkAdmin"
   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${MASTER_SA}" --role "roles/compute.securityAdmin"
   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${MASTER_SA}" --role "roles/iam.serviceAccountUser"
   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${MASTER_SA}" --role "roles/storage.admin"
   
   $ export WORKER_SA=${INFRA_ID}-w@${PROJECT_NAME}.iam.gserviceaccount.com
   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${WORKER_SA}" --role "roles/compute.viewer"
   $ gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member "serviceAccount:${WORKER_SA}" --role "roles/storage.admin"

6. Create a service account key and store it locally for later use:

   $ gcloud iam service-accounts keys create service-account-key.json --iam-account=${MASTER_SA}

Procedure

1. Obtain the RHCOS image from the Product Downloads page on the Red Hat customer portal or the RHCOS image mirror page.

   Important

   The RHCOS images might not change with every release of OpenShift Container Platform. You must download an image with the highest version that is less than or equal to the OpenShift Container Platform version that you install. Use the image version that matches your OpenShift Container Platform version if it is available.

   The file name contains the OpenShift Container Platform version number in the format rhcos-<version>-gcp.tar.

2. Export the following variable:

   $ export IMAGE_SOURCE=<downloaded_image_file_path>

3. Create the cluster image:

   $ gcloud compute images create "${INFRA_ID}-rhcos-image" \
       --source-uri="${IMAGE_SOURCE}"

Procedure

1. Copy the template from the Deployment Manager template for the bootstrap machine section of this topic and save it as 04_bootstrap.py on your computer. This template describes the bootstrap machine that your cluster requires.

2. Export the following variables required by the resource definition:

   $ export CONTROL_SUBNET=`gcloud compute networks subnets describe ${INFRA_ID}-master-subnet --region=${REGION} --format json | jq -r .selfLink`
   $ export CLUSTER_IMAGE=`gcloud compute images describe ${INFRA_ID}-rhcos-image --format json | jq -r .selfLink`
   $ export ZONE_0=`gcloud compute regions describe ${REGION} --format=json | jq -r .zones[0] | cut -d "/" -f9`
   $ export ZONE_1=`gcloud compute regions describe ${REGION} --format=json | jq -r .zones[1] | cut -d "/" -f9`
   $ export ZONE_2=`gcloud compute regions describe ${REGION} --format=json | jq -r .zones[2] | cut -d "/" -f9`

3. Create a bucket and upload the bootstrap.ign file:

   $ gsutil mb gs://${INFRA_ID}-bootstrap-ignition
   $ gsutil cp bootstrap.ign gs://${INFRA_ID}-bootstrap-ignition/

4. Create a signed URL for the bootstrap instance to use to access the Ignition config. Export the URL from the output as a variable:

   $ export BOOTSTRAP_IGN=`gsutil signurl -d 1h service-account-key.json \
       gs://${INFRA_ID}-bootstrap-ignition/bootstrap.ign | grep "^gs:" | awk '{print $5}'`

5. Create a 04_bootstrap.yaml resource definition file:

   $ cat <<EOF >04_bootstrap.yaml
   imports:
   - path: 04_bootstrap.py
   
   resources:
   - name: cluster-bootstrap
     type: 04_bootstrap.py
     properties:
       infra_id: '${INFRA_ID}' 1
       region: '${REGION}' 2
       zone: '${ZONE_0}' 3
   
       cluster_network: '${CLUSTER_NETWORK}' 4
       control_subnet: '${CONTROL_SUBNET}' 5
       image: '${CLUSTER_IMAGE}' 6
       machine_type: 'n1-standard-4' 7
       root_volume_size: '128' 8
   
       bootstrap_ign: '${BOOTSTRAP_IGN}' 9
   EOF

   1

   infra_id is the INFRA_ID infrastructure name from the extraction step.

   2

   region is the region to deploy the cluster into, for example us-east1.

   3

   zone is the zone to deploy the bootstrap instance into, for example us-east1-b.

   4

   cluster_network is the selfLink URL to the cluster network.

   5

   control_subnet is the selfLink URL to the control subnet.

   6

   image is the selfLink URL to the RHCOS image.

   7

   machine_type is the machine type of the instance, for example n1-standard-4.

   8

   bootstrap_ign is the URL output when creating a signed URL above.

6. Create the deployment by using the gcloud CLI:

   $ gcloud deployment-manager deployments create ${INFRA_ID}-bootstrap --config 04_bootstrap.yaml

7. The templates do not manage load balancer membership due to limitations of Deployment Manager, so you must add the bootstrap machine manually:

   $ gcloud compute target-pools add-instances \
       ${INFRA_ID}-api-target-pool --instances-zone="${ZONE_0}" --instances=${INFRA_ID}-bootstrap
   $ gcloud compute target-pools add-instances \
       ${INFRA_ID}-ign-target-pool --instances-zone="${ZONE_0}" --instances=${INFRA_ID}-bootstrap

Procedure

1. Copy the template from the Deployment Manager template for control plane machines section of this topic and save it as 05_control_plane.py on your computer. This template describes the control plane machines that your cluster requires.

2. Export the following variables needed by the resource definition:

   $ export MASTER_SERVICE_ACCOUNT_EMAIL=`gcloud iam service-accounts list | grep "^${INFRA_ID}-master-node " | awk '{print $2}'`
   $ export MASTER_IGNITION=`cat master.ign`

3. Create a 05_control_plane.yaml resource definition file:

   $ cat <<EOF >05_control_plane.yaml
   imports:
   - path: 05_control_plane.py
   
   resources:
   - name: cluster-control-plane
     type: 05_control_plane.py
     properties:
       infra_id: '${INFRA_ID}' 1
       region: '${REGION}' 2
       zones: 3
       - '${ZONE_0}'
       - '${ZONE_1}'
       - '${ZONE_2}'
   
       control_subnet: '${CONTROL_SUBNET}' 4
       image: '${CLUSTER_IMAGE}' 5
       machine_type: 'n1-standard-4' 6
       root_volume_size: '128'
       service_account_email: '${MASTER_SERVICE_ACCOUNT_EMAIL}' 7
   
       ignition: '${MASTER_IGNITION}' 8
   EOF

   1

   infra_id is the INFRA_ID infrastructure name from the extraction step.

   2

   region is the region to deploy the cluster into, for example us-east1.

   3

   zones are the zones to deploy the bootstrap instance into, for example us-east1-b, us-east1-c, and us-east1-d.

   4

   control_subnet is the selfLink URL to the control subnet.

   5

   image is the selfLink URL to the RHCOS image.

   6

   machine_type is the machine type of the instance, for example n1-standard-4.

   7

   service_account_email is the email address for the master service account created above.

   8

   ignition is the contents of the master.ign file.

4. Create the deployment by using the gcloud CLI:

   $ gcloud deployment-manager deployments create ${INFRA_ID}-control-plane --config 05_control_plane.yaml

5. The templates do not manage DNS entries due to limitations of Deployment Manager, so you must add the etcd entries manually:

   $ export MASTER0_IP=`gcloud compute instances describe ${INFRA_ID}-m-0 --zone ${ZONE_0} --format json | jq -r .networkInterfaces[0].networkIP`
   $ export MASTER1_IP=`gcloud compute instances describe ${INFRA_ID}-m-1 --zone ${ZONE_1} --format json | jq -r .networkInterfaces[0].networkIP`
   $ export MASTER2_IP=`gcloud compute instances describe ${INFRA_ID}-m-2 --zone ${ZONE_2} --format json | jq -r .networkInterfaces[0].networkIP`
   $ if [ -f transaction.yaml ]; then rm transaction.yaml; fi
   $ gcloud dns record-sets transaction start --zone ${INFRA_ID}-private-zone
   $ gcloud dns record-sets transaction add ${MASTER0_IP} --name etcd-0.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${INFRA_ID}-private-zone
   $ gcloud dns record-sets transaction add ${MASTER1_IP} --name etcd-1.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${INFRA_ID}-private-zone
   $ gcloud dns record-sets transaction add ${MASTER2_IP} --name etcd-2.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type A --zone ${INFRA_ID}-private-zone
   $ gcloud dns record-sets transaction add \
     "0 10 2380 etcd-0.${CLUSTER_NAME}.${BASE_DOMAIN}." \
     "0 10 2380 etcd-1.${CLUSTER_NAME}.${BASE_DOMAIN}." \
     "0 10 2380 etcd-2.${CLUSTER_NAME}.${BASE_DOMAIN}." \
     --name _etcd-server-ssl._tcp.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 60 --type SRV --zone ${INFRA_ID}-private-zone
   $ gcloud dns record-sets transaction execute --zone ${INFRA_ID}-private-zone

6. The templates do not manage load balancer membership due to limitations of Deployment Manager, so you must add the control plane machines manually:

   $ gcloud compute target-pools add-instances ${INFRA_ID}-api-target-pool --instances-zone="${ZONE_0}" --instances=${INFRA_ID}-m-0
   $ gcloud compute target-pools add-instances ${INFRA_ID}-api-target-pool --instances-zone="${ZONE_1}" --instances=${INFRA_ID}-m-1
   $ gcloud compute target-pools add-instances ${INFRA_ID}-api-target-pool --instances-zone="${ZONE_2}" --instances=${INFRA_ID}-m-2
   $ gcloud compute target-pools add-instances ${INFRA_ID}-ign-target-pool --instances-zone="${ZONE_0}" --instances=${INFRA_ID}-m-0
   $ gcloud compute target-pools add-instances ${INFRA_ID}-ign-target-pool --instances-zone="${ZONE_1}" --instances=${INFRA_ID}-m-1
   $ gcloud compute target-pools add-instances ${INFRA_ID}-ign-target-pool --instances-zone="${ZONE_2}" --instances=${INFRA_ID}-m-2

Procedure

1. Change to the directory that contains the installation program and run the following command:

   $ ./openshift-install wait-for bootstrap-complete --dir=<installation_directory> \ 1
       --log-level info 2

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

   2

   To view different installation details, specify warn, debug, or error instead of info.

   If the command exits without a FATAL warning, your production control plane has initialized.

2. Delete the bootstrap resources:

   $ gcloud compute target-pools remove-instances ${INFRA_ID}-api-target-pool --instances-zone="${ZONE_0}" --instances=${INFRA_ID}-bootstrap
   $ gcloud compute target-pools remove-instances ${INFRA_ID}-ign-target-pool --instances-zone="${ZONE_0}" --instances=${INFRA_ID}-bootstrap
   $ gsutil rm gs://${INFRA_ID}-bootstrap-ignition/bootstrap.ign
   $ gsutil rb gs://${INFRA_ID}-bootstrap-ignition
   $ gcloud deployment-manager deployments delete ${INFRA_ID}-bootstrap

Procedure

1. Copy the template from the Deployment Manager template for worker machines section of this topic and save it as 06_worker.py on your computer. This template describes the worker machines that your cluster requires.

2. Export the following variables needed by the resource definition:

   $ export COMPUTE_SUBNET=`gcloud compute networks subnets describe ${INFRA_ID}-worker-subnet --region=${REGION} --format json | jq -r .selfLink`
   $ export WORKER_SERVICE_ACCOUNT_EMAIL=`gcloud iam service-accounts list | grep "^${INFRA_ID}-worker-node " | awk '{print $2}'`
   $ export WORKER_IGNITION=`cat worker.ign`

3. Create a 06_worker.yaml resource definition file:

   $ cat <<EOF >06_worker.yaml
   imports:
   - path: 06_worker.py
   
   resources:
   - name: 'w-a-0' 1
     type: 06_worker.py
     properties:
       infra_id: '${INFRA_ID}' 2
       region: '${REGION}' 3
       zone: '${ZONE_0}' 4
   
       compute_subnet: '${COMPUTE_SUBNET}' 5
       image: '${CLUSTER_IMAGE}' 6
       machine_type: 'n1-standard-4' 7
       root_volume_size: '128'
       service_account_email: '${WORKER_SERVICE_ACCOUNT_EMAIL}' 8
   
       ignition: '${WORKER_IGNITION}' 9
   EOF

   1

   name is the name of the worker machine, for example w-a-0.

   2

   infra_id is the INFRA_ID infrastructure name from the extraction step.

   3

   region is the region to deploy the cluster into, for example us-east1.

   4

   zone is the zone to deploy the worker machine into, for example us-east1-b.

   5

   compute_subnet is the selfLink URL to the compute subnet.

   6

   image is the selfLink URL to the RHCOS image.

   7

   machine_type is the machine type of the instance, for example n1-standard-4.

   8

   service_account_email is the email address for the worker service account created above.

   9

   ignition is the contents of the worker.ign file.

4. Optional: If you want to launch additional instances, include additional resources of type 06_worker.py in your 06_worker.yaml resource definition file.

5. Create the deployment by using the gcloud CLI:

   $ gcloud deployment-manager deployments create ${INFRA_ID}-worker --config 06_worker.yaml

Procedure

1. From the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site, navigate to the page for your installation type and click Download Command-line Tools.

2. Click the folder for your operating system and architecture and click the compressed file.

   Note

   You can install oc on Linux, Windows, or macOS.

3. Save the file to your file system.
4. Extract the compressed file.
5. Place it in a directory that is on your PATH.

Procedure

1. Export the kubeadmin credentials:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig 1

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

2. Verify you can run oc commands successfully using the exported configuration:

   $ oc whoami
   system:admin

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes
   
   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.14.6+c4799753c
   master-1  Ready     master  63m  v1.14.6+c4799753c
   master-2  Ready     master  64m  v1.14.6+c4799753c
   worker-0  NotReady  worker  76s  v1.14.6+c4799753c
   worker-1  NotReady  worker  70s  v1.14.6+c4799753c

   The output lists all of the machines that you created.

2. Review the pending certificate signing requests (CSRs) and ensure that the you see a client and server request with Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr
   
   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 1
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending 2
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

   1

   A client request CSR.

   2

   A server request CSR.

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. You must implement a method of automatically approving the kubelet serving certificate requests.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • To approve all pending CSRs, run the following command:

     $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve

Procedure

1. Wait for the Ingress router to create a load balancer and populate the EXTERNAL-IP field:

   $ oc -n openshift-ingress get service router-default
   NAME             TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
   router-default   LoadBalancer   172.30.18.154   35.233.157.184   80:32288/TCP,443:31215/TCP   98

2. Add the A record to your public and private zones:

   $ export ROUTER_IP=`oc -n openshift-ingress get service router-default --no-headers | awk '{print $4}'`
   
   $ if [ -f transaction.yaml ]; then rm transaction.yaml; fi
   $ gcloud dns record-sets transaction start --zone ${BASE_DOMAIN_ZONE_NAME}
   $ gcloud dns record-sets transaction add ${ROUTER_IP} --name \*.apps.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 300 --type A --zone ${BASE_DOMAIN_ZONE_NAME}
   $ gcloud dns record-sets transaction execute --zone ${BASE_DOMAIN_ZONE_NAME}
   
   $ if [ -f transaction.yaml ]; then rm transaction.yaml; fi
   $ gcloud dns record-sets transaction start --zone ${INFRA_ID}-private-zone
   $ gcloud dns record-sets transaction add ${ROUTER_IP} --name \*.apps.${CLUSTER_NAME}.${BASE_DOMAIN}. --ttl 300 --type A --zone ${INFRA_ID}-private-zone
   $ gcloud dns record-sets transaction execute --zone ${INFRA_ID}-private-zone

   If you prefer to add explicit domains instead of using a wildcard, you can create entries for each of the cluster’s current routes:

   $ oc get --all-namespaces -o jsonpath='{range .items[*]}{range .status.ingress[*]}{.host}{"\n"}{end}{end}' routes
   oauth-openshift.apps.your.cluster.domain.example.com
   console-openshift-console.apps.your.cluster.domain.example.com
   downloads-openshift-console.apps.your.cluster.domain.example.com
   alertmanager-main-openshift-monitoring.apps.your.cluster.domain.example.com
   grafana-openshift-monitoring.apps.your.cluster.domain.example.com
   prometheus-k8s-openshift-monitoring.apps.your.cluster.domain.example.com

Procedure

1. Complete the cluster installation:

   $ ./openshift-install --dir=<installation_directory> wait-for install-complete 1
   
   INFO Waiting up to 30m0s for the cluster to initialize...

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

   Important

   The Ignition config files that the installation program generates contain certificates that expire after 24 hours. You must keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

2. Observe the running state of your cluster.

   1. Run the following command to view the current cluster version and status:

      $ oc get clusterversion
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version             False       True          24m     Working towards 4.2.0-0: 99% complete

   2. Run the following command to view the Operators managed on the control plane by the Cluster Version Operator (CVO):

      $ oc get clusteroperators
      NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
      authentication                             4.2.0-0   True        False         False      6m18s
      cloud-credential                           4.2.0-0   True        False         False      17m
      cluster-autoscaler                         4.2.0-0   True        False         False      80s
      console                                    4.2.0-0   True        False         False      3m57s
      dns                                        4.2.0-0   True        False         False      22m
      image-registry                             4.2.0-0   True        False         False      5m4s
      ingress                                    4.2.0-0   True        False         False      4m38s
      insights                                   4.2.0-0   True        False         False      21m
      kube-apiserver                             4.2.0-0   True        False         False      12m
      kube-controller-manager                    4.2.0-0   True        False         False      12m
      kube-scheduler                             4.2.0-0   True        False         False      11m
      machine-api                                4.2.0-0   True        False         False      18m
      machine-config                             4.2.0-0   True        False         False      22m
      marketplace                                4.2.0-0   True        False         False      5m38s
      monitoring                                 4.2.0-0   True        False         False      86s
      network                                    4.2.0-0   True        False         False      14m
      node-tuning                                4.2.0-0   True        False         False      6m8s
      openshift-apiserver                        4.2.0-0   True        False         False      6m48s
      openshift-controller-manager               4.2.0-0   True        False         False      12m
      openshift-samples                          4.2.0-0   True        False         False      67s
      operator-lifecycle-manager                 4.2.0-0   True        False         False      15m
      operator-lifecycle-manager-catalog         4.2.0-0   True        False         False      15m
      operator-lifecycle-manager-packageserver   4.2.0-0   True        False         False      6m48s
      service-ca                                 4.2.0-0   True        False         False      17m
      service-catalog-apiserver                  4.2.0-0   True        False         False      6m18s
      service-catalog-controller-manager         4.2.0-0   True        False         False      6m19s
      storage                                    4.2.0-0   True        False         False      6m20s

   3. Run the following command to view your cluster Pods:

      $ oc get pods --all-namespaces
      NAMESPACE                                               NAME                                                                READY     STATUS      RESTARTS   AGE
      kube-system                                             etcd-member-ip-10-0-3-111.us-east-2.compute.internal                1/1       Running     0          35m
      kube-system                                             etcd-member-ip-10-0-3-239.us-east-2.compute.internal                1/1       Running     0          37m
      kube-system                                             etcd-member-ip-10-0-3-24.us-east-2.compute.internal                 1/1       Running     0          35m
      openshift-apiserver-operator                            openshift-apiserver-operator-6d6674f4f4-h7t2t                       1/1       Running     1          37m
      openshift-apiserver                                     apiserver-fm48r                                                     1/1       Running     0          30m
      openshift-apiserver                                     apiserver-fxkvv                                                     1/1       Running     0          29m
      openshift-apiserver                                     apiserver-q85nm                                                     1/1       Running     0          29m
      ...
      openshift-service-ca-operator                           openshift-service-ca-operator-66ff6dc6cd-9r257                      1/1       Running     0          37m
      openshift-service-ca                                    apiservice-cabundle-injector-695b6bcbc-cl5hm                        1/1       Running     0          35m
      openshift-service-ca                                    configmap-cabundle-injector-8498544d7-25qn6                         1/1       Running     0          35m
      openshift-service-ca                                    service-serving-cert-signer-6445fc9c6-wqdqn                         1/1       Running     0          35m
      openshift-service-catalog-apiserver-operator            openshift-service-catalog-apiserver-operator-549f44668b-b5q2w       1/1       Running     0          32m
      openshift-service-catalog-controller-manager-operator   openshift-service-catalog-controller-manager-operator-b78cr2lnm     1/1       Running     0          31m

   When the current cluster version is AVAILABLE, the installation is complete.

Procedure

1. From the computer that you used to install the cluster, run the following command:

   $ ./openshift-install destroy cluster \
   --dir=<installation_directory> --log-level=info 1 2

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

   2

   To view different details, specify warn, debug, or error instead of info.

   Note

   You must specify the directory that contains the cluster definition files for your cluster. The installation program requires the metadata.json file in this directory to delete the cluster.

2. Optional: Delete the <installation_directory> directory and the OpenShift Container Platform installation program.