Installing

Procedure

1. Identify your domain, or subdomain, and registrar. You can transfer an existing domain and registrar or obtain a new one through AWS or another source.

   Note

   If you purchase a new domain through AWS, it takes time for the relevant DNS changes to propagate. For more information about purchasing domains through AWS, see Registering Domain Names Using Amazon Route 53 in the AWS documentation.

2. If you are using an existing domain and registrar, migrate its DNS to AWS. See Making Amazon Route 53 the DNS Service for an Existing Domain in the AWS documentation.

3. Create a public hosted zone for your domain or subdomain. See Creating a Public Hosted Zone in the AWS documentation.

   Use an appropriate root domain, such as openshiftcorp.com, or subdomain, such as clusters.openshiftcorp.com.

4. Extract the new authoritative name servers from the hosted zone records. See Getting the Name Servers for a Public Hosted Zone in the AWS documentation.
5. Update the registrar records for the AWS Route53 name servers that your domain uses. For example, if you registered your domain to a Route53 service in a different accounts, see the following topic in the AWS documentation: Adding or Changing Name Servers or Glue Records.
6. If you use a subdomain, follow your company’s procedures to add its delegation records to the parent domain.

Procedure

1. Specify the IAM user name and select Programmatic access.

2. Attach the AdministratorAccess policy to ensure that the account has sufficient permission to create the cluster. This policy provides the cluster with the ability to grant credentials to each OpenShift Container Platform component. The cluster grants the components only the credentials that they require.

   Note

   While it is possible to create a policy that grants the all of the required AWS permissions and attach it to the user, this is not the preferred option. The cluster will not have the ability to grant additional credentials to individual components, so the same credentials are used by all components.

3. Optional: Add metadata to the user by attaching tags.
4. Confirm that the user name that you specified is granted the AdministratorAccess policy.

5. Record the access key ID and secret access key values. You must use these values when you configure your local machine to run the installation program.

   Important

   You cannot use a temporary session token that you generated while using a multi-factor authentication device to authenticate to AWS when you deploy a cluster. The cluster continues to use your current AWS credentials to create AWS resources for the entire life of the cluster, so you must use key-based, long-lived credentials.

Procedure

1. If you do not have an SSH key that is configured for password-less authentication on your computer, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t rsa -b 4096 -N '' \
       -f <path>/<file_name> 1

   1

   Specify the path and file name, such as ~/.ssh/id_rsa, of the SSH key.

   Running this command generates an SSH key that does not require a password in the location that you specified.

2. Start the ssh-agent process as a background task:

   $ eval "$(ssh-agent -s)"
   
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> 1
   
   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   1

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_rsa

Procedure

1. Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

2. Navigate to the page for your installation type, download the installation program for your operating system, and place the file in the directory where you will store the installation configuration files.

   Important

   The installation program creates several files on the computer that you use to install your cluster. You must keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar xvf <installation_program>.tar.gz

4. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret as a .txt file or copy it to your clipboard. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

Procedure

1. Run the installation program:

   $ ./openshift-install create cluster --dir=<installation_directory> \ 1
       --log-level info 2

   1

   For <installation_directory>, specify the directory name to store the files that the installation program creates.

   2

   To view different installation details, specify warn, debug, or error instead of info.

   Important

   Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

   Provide values at the prompts:

   1. Optional: Select an SSH key to use to access your cluster machines.

      Note

      For production OpenShift Container Platform clusters on which you want to perform installation debugging or disaster recovery, you must provide an SSH key that your ssh-agent process uses to the installation program.

   2. Select AWS as the platform to target.
   3. If you do not have an Amazon Web Services (AWS) profile stored on your computer, enter the AWS access key ID and secret access key for the user that you configured to run the installation program.
   4. Select the AWS region to deploy the cluster to.
   5. Select the base domain for the Route53 service that you configured for your cluster.
   6. Enter a descriptive name for your cluster.
   7. Paste the pull secret that you obtained from the Pull Secret page on the Red Hat OpenShift Cluster Manager site.
   Note

   If the cloud provider account that you configured on your host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed.

   When the cluster deployment completes, directions for accessing your cluster, including a link to its web console and credentials for the kubeadmin user, display in your terminal.

   Important

   The Ignition config files that the installation program generates contain certificates that expire after 24 hours. You must keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

   Important

   You must not delete the installation program or the files that the installation program creates. Both are required to delete the cluster.

2. Optional: Remove or disable the AdministratorAccess policy from the IAM account that you used to install the cluster.

Procedure

1. From the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site, navigate to the page for your installation type and click Download Command-line Tools.

2. From the site that is displayed, download the compressed file for your operating system.

   Note

   You can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Procedure

1. Export the kubeadmin credentials:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig 1
   
   $ oc whoami
   system:admin

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

Procedure

1. If you do not have an SSH key that is configured for password-less authentication on your computer, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t rsa -b 4096 -N '' \
       -f <path>/<file_name> 1

   1

   Specify the path and file name, such as ~/.ssh/id_rsa, of the SSH key.

   Running this command generates an SSH key that does not require a password in the location that you specified.

2. Start the ssh-agent process as a background task:

   $ eval "$(ssh-agent -s)"
   
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> 1
   
   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   1

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_rsa

Procedure

1. Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

2. Navigate to the page for your installation type, download the installation program for your operating system, and place the file in the directory where you will store the installation configuration files.

   Important

   The installation program creates several files on the computer that you use to install your cluster. You must keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar xvf <installation_program>.tar.gz

4. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret as a .txt file or copy it to your clipboard. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

Procedure

1. Create the install-config.yaml file.

   1. Run the following command:

      $ ./openshift-install create install-config --dir=<installation_directory> 1

      1

      For <installation_directory>, specify the directory name to store the files that the installation program creates.

      Important

      Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

   2. At the prompts, provide the configuration details for your cloud:

      1. Optional: Select an SSH key to use to access your cluster machines.

         Note

         For production OpenShift Container Platform clusters on which you want to perform installation debugging or disaster recovery, you must provide an SSH key that your ssh-agent process uses to the installation program.

      2. Select AWS as the platform to target.
      3. If you do not have an Amazon Web Services (AWS) profile stored on your computer, enter the AWS access key ID and secret access key for the user that you configured to run the installation program.
      4. Select the AWS region to deploy the cluster to.
      5. Select the base domain for the Route53 service that you configured for your cluster.
      6. Enter a descriptive name for your cluster.
      7. Paste the pull secret that you obtained from the Pull Secret page on the Red Hat OpenShift Cluster Manager site.
2. Modify the install-config.yaml file. You can find more information about the available parameters in the Installation configuration parameters section and in the Go documentation.

3. Back up the install-config.yaml file so that you can use it to install multiple clusters.

   Important

   The install-config.yaml file is consumed during the installation process. If you want to reuse the file, you must back it up now.

Procedure

1. Run the installation program:

   $ ./openshift-install create cluster --dir=<installation_directory> \ 1
       --log-level info 2

   1

   For <installation_directory>, specify the location of your customized ./install-config.yaml file.

   2

   For <installation_directory>, specify the directory name to store the files that the installation program creates.

   To view different installation details, specify warn, debug, or error instead of info.

   Important

   Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

   Provide values at the prompts:

   1. Optional: Select an SSH key to use to access your cluster machines.

      Note

      For production OpenShift Container Platform clusters on which you want to perform installation debugging or disaster recovery, you must provide an SSH key that your ssh-agent process uses to the installation program.

   2. Select AWS as the platform to target.
   3. If you do not have an Amazon Web Services (AWS) profile stored on your computer, enter the AWS access key ID and secret access key for the user that you configured to run the installation program.
   4. Select the AWS region to deploy the cluster to.
   5. Select the base domain for the Route53 service that you configured for your cluster.
   6. Enter a descriptive name for your cluster.
   7. Paste the pull secret that you obtained from the Pull Secret page on the Red Hat OpenShift Cluster Manager site.
   Note

   If the cloud provider account that you configured on your host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed.

   When the cluster deployment completes, directions for accessing your cluster, including a link to its web console and credentials for the kubeadmin user, display in your terminal.

   Important

   The Ignition config files that the installation program generates contain certificates that expire after 24 hours. You must keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

   Important

   You must not delete the installation program or the files that the installation program creates. Both are required to delete the cluster.

2. Optional: Remove or disable the AdministratorAccess policy from the IAM account that you used to install the cluster.

Procedure

1. From the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site, navigate to the page for your installation type and click Download Command-line Tools.

2. From the site that is displayed, download the compressed file for your operating system.

   Note

   You can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Procedure

1. Export the kubeadmin credentials:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig 1
   
   $ oc whoami
   system:admin

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

Procedure

1. If you do not have an SSH key that is configured for password-less authentication on your computer, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t rsa -b 4096 -N '' \
       -f <path>/<file_name> 1

   1

   Specify the path and file name, such as ~/.ssh/id_rsa, of the SSH key.

   Running this command generates an SSH key that does not require a password in the location that you specified.

2. Start the ssh-agent process as a background task:

   $ eval "$(ssh-agent -s)"
   
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> 1
   
   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   1

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_rsa

Procedure

1. Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

2. Navigate to the page for your installation type, download the installation program for your operating system, and place the file in the directory where you will store the installation configuration files.

   Important

   The installation program creates several files on the computer that you use to install your cluster. You must keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar xvf <installation_program>.tar.gz

4. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret as a .txt file or copy it to your clipboard. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

Procedure

1. Create the install-config.yaml file.

   1. Run the following command:

      $ ./openshift-install create install-config --dir=<installation_directory> 1

      1

      For <installation_directory>, specify the directory name to store the files that the installation program creates.

      Important

      Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

   2. At the prompts, provide the configuration details for your cloud:

      1. Optional: Select an SSH key to use to access your cluster machines.

         Note

         For production OpenShift Container Platform clusters on which you want to perform installation debugging or disaster recovery, you must provide an SSH key that your ssh-agent process uses to the installation program.

      2. Select AWS as the platform to target.
      3. If you do not have an Amazon Web Services (AWS) profile stored on your computer, enter the AWS access key ID and secret access key for the user that you configured to run the installation program.
      4. Select the AWS region to deploy the cluster to.
      5. Select the base domain for the Route53 service that you configured for your cluster.
      6. Enter a descriptive name for your cluster.
      7. Paste the pull secret that you obtained from the Pull Secret page on the Red Hat OpenShift Cluster Manager site.
2. Modify the install-config.yaml file. You can find more information about the available parameters in the Installation configuration parameters section and in the Go documentation.

3. Back up the install-config.yaml file so that you can use it to install multiple clusters.

   Important

   The install-config.yaml file is consumed during the installation process. If you want to reuse the file, you must back it up now.

Procedure

1. Use the following command to create manifests:

   $ ./openshift-install create manifests --dir=<installation_directory> 1

   1

   For <installation_directory>, specify the name of the directory that contains the install-config.yaml file for your cluster.

2. Create a file that is named cluster-network-03-config.yml in the <installation_directory>/manifests/ directory:

   $ touch <installation_directory>/manifests/cluster-network-03-config.yml 1

   1

   For <installation_directory>, specify the directory name that contains the manifests/ directory for your cluster.

   After creating the file, three network configuration files are in the manifests/ directory, as shown:

   $ ls <installation_directory>/manifests/cluster-network-*
   cluster-network-01-crd.yml
   cluster-network-02-config.yml
   cluster-network-03-config.yml

3. Open the cluster-network-03-config.yml file in an editor and enter a CR that describes the Operator configuration you want:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec: 1
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     serviceNetwork:
     - 172.30.0.0/16
     defaultNetwork:
       type: OpenShiftSDN
       openshiftSDNConfig:
         mode: NetworkPolicy
         mtu: 1450
         vxlanPort: 4789

   1

   The parameters for the spec field are only an example. Specify your configuration for the Network Operator in the CR.

   The Network Operator provides default values for the parameters in the CR, so you must specify only the parameters that you want to change in the Network.operator.openshift.io CR.

4. Save the cluster-network-03-config.yml file and quit the text editor.
5. Optional: Back up the manifests/cluster-network-03-config.yml file. The installation program deletes the manifests/ directory when creating the cluster.

Procedure

1. Run the installation program:

   $ ./openshift-install create cluster --dir=<installation_directory> \ 1
       --log-level info 2

   1

   For <installation_directory>, specify the location of your customized ./install-config.yaml file.

   2

   For <installation_directory>, specify the directory name to store the files that the installation program creates.

   To view different installation details, specify warn, debug, or error instead of info.

   Important

   Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

   Provide values at the prompts:

   1. Optional: Select an SSH key to use to access your cluster machines.

      Note

      For production OpenShift Container Platform clusters on which you want to perform installation debugging or disaster recovery, you must provide an SSH key that your ssh-agent process uses to the installation program.

   2. Select AWS as the platform to target.
   3. If you do not have an Amazon Web Services (AWS) profile stored on your computer, enter the AWS access key ID and secret access key for the user that you configured to run the installation program.
   4. Select the AWS region to deploy the cluster to.
   5. Select the base domain for the Route53 service that you configured for your cluster.
   6. Enter a descriptive name for your cluster.
   7. Paste the pull secret that you obtained from the Pull Secret page on the Red Hat OpenShift Cluster Manager site.
   Note

   If the cloud provider account that you configured on your host does not have sufficient permissions to deploy the cluster, the installation process stops, and the missing permissions are displayed.

   When the cluster deployment completes, directions for accessing your cluster, including a link to its web console and credentials for the kubeadmin user, display in your terminal.

   Important

   The Ignition config files that the installation program generates contain certificates that expire after 24 hours. You must keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

   Important

   You must not delete the installation program or the files that the installation program creates. Both are required to delete the cluster.

2. Optional: Remove or disable the AdministratorAccess policy from the IAM account that you used to install the cluster.

Procedure

1. From the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site, navigate to the page for your installation type and click Download Command-line Tools.

2. From the site that is displayed, download the compressed file for your operating system.

   Note

   You can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Procedure

1. Export the kubeadmin credentials:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig 1
   
   $ oc whoami
   system:admin

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

Procedure

1. From the computer that you used to install the cluster, run the following command:

   $ ./openshift-install destroy cluster \
   --dir=<installation_directory> --log-level=info 1 2

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

   2

   To view different details, specify warn, debug, or error instead of info.

   Note

   You must specify the directory that contains the cluster definition files for your cluster. The installation program requires the metadata.json file in this directory to delete the cluster.

2. Optional: Delete the <installation_directory> directory and the OpenShift Container Platform installation program.

Procedure

1. Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

2. Navigate to the page for your installation type, download the installation program for your operating system, and place the file in the directory where you will store the installation configuration files.

   Important

   The installation program creates several files on the computer that you use to install your cluster. You must keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar xvf <installation_program>.tar.gz

4. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret as a .txt file or copy it to your clipboard. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

Procedure

1. If you do not have an SSH key that is configured for password-less authentication on your computer, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t rsa -b 4096 -N '' \
       -f <path>/<file_name> 1

   1

   Specify the path and file name, such as ~/.ssh/id_rsa, of the SSH key.

   Running this command generates an SSH key that does not require a password in the location that you specified.

2. Start the ssh-agent process as a background task:

   $ eval "$(ssh-agent -s)"
   
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> 1
   
   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   1

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_rsa

Procedure

1. Obtain the install-config.yaml file.

   1. Run the following command:

      $ ./openshift-install create install-config --dir=<installation_directory> 1

      1

      For <installation_directory>, specify the directory name to store the files that the installation program creates.

      Important

      Specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

   2. At the prompts, provide the configuration details for your cloud:

      1. Optional: Select an SSH key to use to access your cluster machines.

         Note

         For production OpenShift Container Platform clusters on which you want to perform installation debugging or disaster recovery, you must provide an SSH key that your ssh-agent process uses to the installation program.

      2. Select AWS as the platform to target.
      3. If you do not have an AWS profile stored on your computer, enter the AWS access key ID and secret access key for the user that you configured to run the installation program.
      4. Select the AWS region to deploy the cluster to.
      5. Select the base domain for the Route53 service that you configured for your cluster.
      6. Enter a descriptive name for your cluster.
      7. Paste the pull secret that you obtained from the Pull Secret page on the Red Hat OpenShift Cluster Manager site.

2. Edit the install-config.yaml file to set the number of compute, or worker, replicas to 0, as shown in the following compute stanza:

   compute:
   - hyperthreading: Enabled
     name: worker
     platform: {}
     replicas: 0

3. Optional: Back up the install-config.yaml file.

   Important

   The install-config.yaml file is consumed during the next step. If you want to reuse the file, back it up now.

4. Remove the Kubernetes manifest files for the control plane machines. By removing these files, you prevent the cluster from automatically generating control plane machines.

   1. Generate the Kubernetes manifests for the cluster:

      $ ./openshift-install create manifests --dir=<installation_directory> 1
      
      WARNING There are no compute nodes specified. The cluster will not fully initialize without compute nodes.
      INFO Consuming "Install Config" from target directory

      1

      For <installation_directory>, specify the same installation directory.

      Because you create your own compute machines later in the installation process, you can safely ignore this warning.

   2. Remove the files that define the control plane machines:

      $ rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml

5. Remove the Kubernetes manifest files that define the worker machines:

   $ rm -f openshift/99_openshift-cluster-api_worker-machineset-*

   Because you create and manage the worker machines yourself, you do not need to initialize these machines.

6. Obtain the Ignition config files:

   $ ./openshift-install create ignition-configs --dir=<installation_directory> 1

   1

   For <installation_directory>, specify the same installation directory.

   The following files are generated in the directory:

   .
   ├── auth
   │   ├── kubeadmin-password
   │   └── kubeconfig
   ├── bootstrap.ign
   ├── master.ign
   ├── metadata.json
   └── worker.ign

Procedure

1. Create a JSON file that contains the parameter values that the template requires:

   [
     {
       "ParameterKey": "VpcCidr", 1
       "ParameterValue": "10.0.0.0/16" 2
     },
     {
       "ParameterKey": "AvailabilityZoneCount", 3
       "ParameterValue": "1" 4
     },
     {
       "ParameterKey": "SubnetBits", 5
       "ParameterValue": "12" 6
     }
   ]

   1

   The CIDR block for the VPC.

   2

   Specify a CIDR block in the format x.x.x.x/16-24.

   3

   The number of availability zones to deploy the VPC in.

   4

   Specify an integer between 1 and 3.

   5

   The size of each subnet in each availability zone.

   6

   Specify an integer between 5 and 13, where 5 is /27 and 13 is /19.

2. Copy the template from the CloudFormation template for the VPC section of this topic and save it as a YAML file on your computer. This template describes the VPC that your cluster requires.

3. Launch the template:

   Important

   You must enter the command on a single line.

   $ aws cloudformation create-stack --stack-name <name> 1
        --template-body file://<template>.yaml 2
        --parameters file://<parameters>.json 3

   1

   <name> is the name for the CloudFormation stack, such as cluster-vpc. You need the name of this stack if you remove the cluster.

   2

   <template> is the relative path to and name of the CloudFormation template YAML file that you saved.

   3

   <parameters> is the relative path to and name of the CloudFormation parameters JSON file.

4. Confirm that the template components exist:

   $ aws cloudformation describe-stacks --stack-name <name>

   After the StackStatus displays CREATE_COMPLETE, the output displays values for the following parameters. You must provide these parameter values to the other CloudFormation templates that you run to create your cluster:

   VpcId	The ID of your VPC.
   PublicSubnetIds	The IDs of the new public subnets.
   PrivateSubnetIds	The IDs of the new private subnets.

Procedure

1. Obtain the Hosted Zone ID for the Route53 zone that you specified in the install-config.yaml file for your cluster. You can obtain this ID from the AWS console or by running the following command:

   Important

   You must enter the command on a single line.

   $ aws route53 list-hosted-zones-by-name |
        jq --arg name "<route53_domain>." \ 1
        -r '.HostedZones | .[] | select(.Name=="\($name)") | .Id'

   1

   For the <route53_domain>, specify the Route53 base domain that you used when you generated the install-config.yaml file for the cluster.

2. Create a JSON file that contains the parameter values that the template requires:

   [
     {
       "ParameterKey": "ClusterName", 1
       "ParameterValue": "mycluster" 2
     },
     {
       "ParameterKey": "InfrastructureName", 3
       "ParameterValue": "mycluster-<random_string>" 4
     },
     {
       "ParameterKey": "HostedZoneId", 5
       "ParameterValue": "<random_string>" 6
     },
     {
       "ParameterKey": "HostedZoneName", 7
       "ParameterValue": "example.com" 8
     },
     {
       "ParameterKey": "PublicSubnets", 9
       "ParameterValue": "subnet-<random_string>" 10
     },
     {
       "ParameterKey": "PrivateSubnets", 11
       "ParameterValue": "subnet-<random_string>" 12
     },
     {
       "ParameterKey": "VpcId", 13
       "ParameterValue": "vpc-<random_string>" 14
     }
   ]

   1

   A short, representative cluster name to use for host names, etc.

   2

   Specify the cluster name that you used when you generated the install-config.yaml file for the cluster.

   3

   The name for your cluster infrastructure that is encoded in your Ignition config files for the cluster.

   4

   Specify the infrastructure name that you extracted from the Ignition config file metadata, which has the format <cluster-name>-<random-string>.

   5

   The Route53 public zone ID to register the targets with.

   6

   Specify the Route53 public zone ID, which as a format similar to Z21IXYZABCZ2A4. You can obtain this value from the AWS console.

   7

   The Route53 zone to register the targets with.

   8

   Specify the Route53 base domain that you used when you generated the install-config.yaml file for the cluster. Do not include the trailing period (.) that is displayed in the AWS console.

   9

   The public subnets that you created for your VPC.

   10

   Specify the PublicSubnetIds value from the output of the CloudFormation template for the VPC.

   11

   The private subnets that you created for your VPC.

   12

   Specify the PrivateSubnetIds value from the output of the CloudFormation template for the VPC.

   13

   The VPC that you created for the cluster.

   14

   Specify the VpcId value from the output of the CloudFormation template for the VPC.

3. Copy the template from the CloudFormation template for the network and load balancers section of this topic and save it as a YAML file on your computer. This template describes the networking and load balancing objects that your cluster requires.

4. Launch the template:

   Important

   You must enter the command on a single line.

   $ aws cloudformation create-stack --stack-name <name> 1
        --template-body file://<template>.yaml 2
        --parameters file://<parameters>.json 3
        --capabilities CAPABILITY_NAMED_IAM

   1

   <name> is the name for the CloudFormation stack, such as cluster-dns. You need the name of this stack if you remove the cluster.

   2

   <template> is the relative path to and name of the CloudFormation template YAML file that you saved.

   3

   <parameters> is the relative path to and name of the CloudFormation parameters JSON file.

5. Confirm that the template components exist:

   $ aws cloudformation describe-stacks --stack-name <name>

   After the StackStatus displays CREATE_COMPLETE, the output displays values for the following parameters. You must provide these parameter values to the other CloudFormation templates that you run to create your cluster:

   PrivateHostedZoneId	Hosted zone ID for the private DNS.
   ExternalApiLoadBalancerName	Full name of the external API load balancer.
   InternalApiLoadBalancerName	Full name of the internal API load balancer.
   ApiServerDnsName	Full host name of the API server.
   RegisterNlbIpTargetsLambda	Lambda ARN useful to help register/deregister IP targets for these load balancers.
   ExternalApiTargetGroupArn	ARN of external API target group.
   InternalApiTargetGroupArn	ARN of internal API target group.
   InternalServiceTargetGroupArn	ARN of internal service target group.

Procedure

1. Create a JSON file that contains the parameter values that the template requires:

   [
     {
       "ParameterKey": "InfrastructureName", 1
       "ParameterValue": "mycluster-<random_string>" 2
     },
     {
       "ParameterKey": "VpcCidr", 3
       "ParameterValue": "10.0.0.0/16" 4
     },
     {
       "ParameterKey": "PrivateSubnets", 5
       "ParameterValue": "subnet-<random_string>" 6
     },
     {
       "ParameterKey": "VpcId", 7
       "ParameterValue": "vpc-<random_string>" 8
     }
   ]

   1

   The name for your cluster infrastructure that is encoded in your Ignition config files for the cluster.

   2

   Specify the infrastructure name that you extracted from the Ignition config file metadata, which has the format <cluster-name>-<random-string>.

   3

   The CIDR block for the VPC.

   4

   Specify the CIDR block parameter that you used for the VPC that you defined in the form x.x.x.x/16-24.

   5

   The private subnets that you created for your VPC.

   6

   Specify the PrivateSubnetIds value from the output of the CloudFormation template for the VPC.

   7

   The VPC that you created for the cluster.

   8

   Specify the VpcId value from the output of the CloudFormation template for the VPC.

2. Copy the template from the CloudFormation template for security objects section of this topic and save it as a YAML file on your computer. This template describes the security groups and roles that your cluster requires.

3. Launch the template:

   Important

   You must enter the command on a single line.

   $ aws cloudformation create-stack --stack-name <name> 1
        --template-body file://<template>.yaml 2
        --parameters file://<parameters>.json 3
        --capabilities CAPABILITY_NAMED_IAM

   1

   <name> is the name for the CloudFormation stack, such as cluster-sec. You need the name of this stack if you remove the cluster.

   2

   <template> is the relative path to and name of the CloudFormation template YAML file that you saved.

   3

   <parameters> is the relative path to and name of the CloudFormation parameters JSON file.

4. Confirm that the template components exist:

   $ aws cloudformation describe-stacks --stack-name <name>

   After the StackStatus displays CREATE_COMPLETE, the output displays values for the following parameters. You must provide these parameter values to the other CloudFormation templates that you run to create your cluster:

   MasterSecurityGroupId	Master Security Group ID
   WorkerSecurityGroupId	Worker Security Group ID
   MasterInstanceProfile	Master IAM Instance Profile
   WorkerInstanceProfile	Worker IAM Instance Profile

Procedure

1. Provide a location to serve the bootstrap.ign Ignition config file to your cluster. This file is located in your installation directory. One way to do this is to create an S3 bucket in your cluster’s region and upload the Ignition config file to it.

   Important

   The provided CloudFormation Template assumes that the Ignition config files for your cluster are served from an S3 bucket. If you choose to serve the files from another location, you must modify the templates.

   Note

   The bootstrap Ignition config file does contain secrets, like X.509 keys. The following steps provide basic security for the S3 bucket. To provide additional security, you can enable an S3 bucket policy to allow only certain users, such as the OpenShift IAM user, to access objects that the bucket contains. You can avoid S3 entirely and serve your bootstrap Ignition config file from any address that the bootstrap machine can reach.

   1. Create the bucket:

      $ aws s3 mb s3://<cluster-name>-infra 1

      1

      <cluster-name>-infra is the bucket name.

   2. Upload the bootstrap.ign Ignition config file to the bucket:

      $ aws s3 cp bootstrap.ign s3://<cluster-name>-infra/bootstrap.ign

   3. Verify that the file uploaded:

      $ aws s3 ls s3://<cluster-name>-infra/
      
      2019-04-03 16:15:16     314878 bootstrap.ign

2. Create a JSON file that contains the parameter values that the template requires:

   [
     {
       "ParameterKey": "InfrastructureName", 1
       "ParameterValue": "mycluster-<random_string>" 2
     },
     {
       "ParameterKey": "RhcosAmi", 3
       "ParameterValue": "ami-<random_string>" 4
     },
     {
       "ParameterKey": "AllowedBootstrapSshCidr", 5
       "ParameterValue": "0.0.0.0/0" 6
     },
     {
       "ParameterKey": "PublicSubnet", 7
       "ParameterValue": "subnet-<random_string>" 8
     },
     {
       "ParameterKey": "MasterSecurityGroupId", 9
       "ParameterValue": "sg-<random_string>" 10
     },
     {
       "ParameterKey": "VpcId", 11
       "ParameterValue": "vpc-<random_string>" 12
     },
     {
       "ParameterKey": "BootstrapIgnitionLocation", 13
       "ParameterValue": "s3://<bucket_name>/bootstrap.ign" 14
     },
     {
       "ParameterKey": "AutoRegisterELB", 15
       "ParameterValue": "yes" 16
     },
     {
       "ParameterKey": "RegisterNlbIpTargetsLambdaArn", 17
       "ParameterValue": "arn:aws:lambda:<region>:<account_number>:function:<dns_stack_name>-RegisterNlbIpTargets-<random_string>" 18
     },
     {
       "ParameterKey": "ExternalApiTargetGroupArn", 19
       "ParameterValue": "arn:aws:elasticloadbalancing:<region>:<account_number>:targetgroup/<dns_stack_name>-Exter-<random_string>" 20
     },
     {
       "ParameterKey": "InternalApiTargetGroupArn", 21
       "ParameterValue": "arn:aws:elasticloadbalancing:<region>:<account_number>:targetgroup/<dns_stack_name>-Inter-<random_string>" 22
     },
     {
       "ParameterKey": "InternalServiceTargetGroupArn", 23
       "ParameterValue": "arn:aws:elasticloadbalancing:<region>:<account_number>:targetgroup/<dns_stack_name>-Inter-<random_string>" 24
     }
   ]

   1

   The name for your cluster infrastructure that is encoded in your Ignition config files for the cluster.

   2

   Specify the infrastructure name that you extracted from the Ignition config file metadata, which has the format <cluster-name>-<random-string>.

   3

   Current Red Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the bootstrap node.

   4

   Specify a valid AWS::EC2::Image::Id value.

   5

   CIDR block to allow SSH access to the bootstrap node.

   6

   Specify a CIDR block in the format x.x.x.x/16-24.

   7

   The public subnet that is associated with your VPC to launch the bootstrap node into.

   8

   Specify the PublicSubnetIds value from the output of the CloudFormation template for the VPC.

   9

   The master security group ID (for registering temporary rules)

   10

   Specify the MasterSecurityGroupId value from the output of the CloudFormation template for the security group and roles.

   11

   The VPC created resources will belong to.

   12

   Specify the VpcId value from the output of the CloudFormation template for the VPC.

   13

   Location to fetch bootstrap Ignition config file from.

   14

   Specify the S3 bucket and file name in the form s3://<bucket_name>/bootstrap.ign.

   15

   Whether or not to register a network load balancer (NLB).

   16

   Specify yes or no. If you specify yes, you must provide a Lambda Amazon Resource Name (ARN) value.

   17

   The ARN for NLB IP target registration lambda group.

   18

   Specify the RegisterNlbIpTargetsLambda value from the output of the CloudFormation template for DNS and load balancing.

   19

   The ARN for external API load balancer target group.

   20

   Specify the ExternalApiTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

   21

   The ARN for internal API load balancer target group.

   22

   Specify the InternalApiTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

   23

   The ARN for internal service load balancer target group.

   24

   Specify the InternalServiceTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

3. Copy the template from the CloudFormation template for the bootstrap machine section of this topic and save it as a YAML file on your computer. This template describes the bootstrap machine that your cluster requires.

4. Launch the template:

   Important

   You must enter the command on a single line.

   $ aws cloudformation create-stack --stack-name <name> 1
        --template-body file://<template>.yaml 2
        --parameters file://<parameters>.json 3
        --capabilities CAPABILITY_NAMED_IAM

   1

   <name> is the name for the CloudFormation stack, such as cluster-bootstrap. You need the name of this stack if you remove the cluster.

   2

   <template> is the relative path to and name of the CloudFormation template YAML file that you saved.

   3

   <parameters> is the relative path to and name of the CloudFormation parameters JSON file.

5. Confirm that the template components exist:

   $ aws cloudformation describe-stacks --stack-name <name>

   After the StackStatus displays CREATE_COMPLETE, the output displays values for the following parameters. You must provide these parameter values to the other CloudFormation templates that you run to create your cluster:

   BootstrapInstanceId	The bootstrap Instance ID.
   BootstrapPublicIp	The bootstrap node public IP address.
   BootstrapPrivateIp	The bootstrap node private IP address.

Procedure

1. Create a JSON file that contains the parameter values that the template requires:

   [
     {
       "ParameterKey": "InfrastructureName", 1
       "ParameterValue": "mycluster-<random_string>" 2
     },
     {
       "ParameterKey": "RhcosAmi", 3
       "ParameterValue": "ami-<random_string>" 4
     },
     {
       "ParameterKey": "AutoRegisterDNS", 5
       "ParameterValue": "yes" 6
     },
     {
       "ParameterKey": "PrivateHostedZoneId", 7
       "ParameterValue": "<random_string>" 8
     },
     {
       "ParameterKey": "PrivateHostedZoneName", 9
       "ParameterValue": "mycluster.example.com" 10
     },
     {
       "ParameterKey": "Master0Subnet", 11
       "ParameterValue": "subnet-<random_string>" 12
     },
     {
       "ParameterKey": "Master1Subnet", 13
       "ParameterValue": "subnet-<random_string>" 14
     },
     {
       "ParameterKey": "Master2Subnet", 15
       "ParameterValue": "subnet-<random_string>" 16
     },
     {
       "ParameterKey": "MasterSecurityGroupId", 17
       "ParameterValue": "sg-<random_string>" 18
     },
     {
       "ParameterKey": "IgnitionLocation", 19
       "ParameterValue": "https://api-int.<cluster_name>.<domain_name>:22623/config/master" 20
     },
     {
       "ParameterKey": "CertificateAuthorities", 21
       "ParameterValue": "data:text/plain;charset=utf-8;base64,ABC...xYz==" 22
     },
     {
       "ParameterKey": "MasterInstanceProfileName", 23
       "ParameterValue": "<roles_stack>-MasterInstanceProfile-<random_string>" 24
     },
     {
       "ParameterKey": "MasterInstanceType", 25
       "ParameterValue": "m4.xlarge" 26
     },
     {
       "ParameterKey": "AutoRegisterELB", 27
       "ParameterValue": "yes" 28
     },
     {
       "ParameterKey": "RegisterNlbIpTargetsLambdaArn", 29
       "ParameterValue": "arn:aws:lambda:<region>:<account_number>:function:<dns_stack_name>-RegisterNlbIpTargets-<random_string>" 30
     },
     {
       "ParameterKey": "ExternalApiTargetGroupArn", 31
       "ParameterValue": "arn:aws:elasticloadbalancing:<region>:<account_number>:targetgroup/<dns_stack_name>-Exter-<random_string>" 32
     },
     {
       "ParameterKey": "InternalApiTargetGroupArn", 33
       "ParameterValue": "arn:aws:elasticloadbalancing:<region>:<account_number>:targetgroup/<dns_stack_name>-Inter-<random_string>" 34
     },
     {
       "ParameterKey": "InternalServiceTargetGroupArn", 35
       "ParameterValue": "arn:aws:elasticloadbalancing:<region>:<account_number>:targetgroup/<dns_stack_name>-Inter-<random_string>" 36
     }
   ]

   1

   The name for your cluster infrastructure that is encoded in your Ignition config files for the cluster.

   2

   Specify the infrastructure name that you extracted from the Ignition config file metadata, which has the format <cluster-name>-<random-string>.

   3

   CurrentRed Hat Enterprise Linux CoreOS (RHCOS) AMI to use for the control plane machines.

   4

   Specify an AWS::EC2::Image::Id value.

   5

   Whether or not to perform DNS etcd registration.

   6

   Specify yes or no. If you specify yes, you must provide Hosted Zone information.

   7

   The Route53 private zone ID to register the etcd targets with.

   8

   Specify the PrivateHostedZoneId value from the output of the CloudFormation template for DNS and load balancing.

   9

   The Route53 zone to register the targets with.

   10

   Specify <cluster_name>.<domain_name> where <domain_name> is the Route53 base domain that you used when you generated install-config.yaml file for the cluster. Do not include the trailing period (.) that is displayed in the AWS console.

   11 13 15

   A subnet, preferably private, to launch the control plane machines on.

   12 14 16

   Specify a subnet from the PrivateSubnets value from the output of the CloudFormation template for DNS and load balancing.

   17

   The master security group ID to associate with master nodes.

   18

   Specify the MasterSecurityGroupId value from the output of the CloudFormation template for the security group and roles.

   19

   The location to fetch control plane Ignition config file from.

   20

   Specify the generated Ignition config file location, https://api-int.<cluster_name>.<domain_name>:22623/config/master.

   21

   The base64 encoded certificate authority string to use.

   22

   Specify the value from the master.ign file that is in the installation directory. This value is the long string with the format data:text/plain;charset=utf-8;base64,ABC…​xYz==.

   23

   The IAM profile to associate with master nodes.

   24

   Specify the MasterInstanceProfile parameter value from the output of the CloudFormation template for the security group and roles.

   25

   The type of AWS instance to use for the control plane machines.

   26

   Allowed values:

   • m4.xlarge
   • m4.2xlarge
   • m4.4xlarge
   • m4.8xlarge
   • m4.10xlarge
   • m4.16xlarge
   • c4.2xlarge
   • c4.4xlarge
   • c4.8xlarge
   • r4.xlarge
   • r4.2xlarge
   • r4.4xlarge
   • r4.8xlarge

   • r4.16xlarge

     Important

     If m4 instance types are not available in your region, such as with eu-west-3, specify an m5 type, such as m5.xlarge, instead.

   27

   Whether or not to register a network load balancer (NLB).

   28

   Specify yes or no. If you specify yes, you must provide a Lambda Amazon Resource Name (ARN) value.

   29

   The ARN for NLB IP target registration lambda group.

   30

   Specify the RegisterNlbIpTargetsLambda value from the output of the CloudFormation template for DNS and load balancing.

   31

   The ARN for external API load balancer target group.

   32

   Specify the ExternalApiTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

   33

   The ARN for internal API load balancer target group.

   34

   Specify the InternalApiTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

   35

   The ARN for internal service load balancer target group.

   36

   Specify the InternalServiceTargetGroupArn value from the output of the CloudFormation template for DNS and load balancing.

2. Copy the template from the CloudFormation template for control plane machines section of this topic and save it as a YAML file on your computer. This template describes the control plane machines that your cluster requires.
3. If you specified an m5 instance type as the value for MasterInstanceType, add that instance type to the MasterInstanceType.AllowedValues parameter in the CloudFormation template.

4. Launch the template:

   Important

   You must enter the command on a single line.

   $ aws cloudformation create-stack --stack-name <name> 1
        --template-body file://<template>.yaml 2
        --parameters file://<parameters>.json 3

   1

   <name> is the name for the CloudFormation stack, such as cluster-control-plane. You need the name of this stack if you remove the cluster.

   2

   <template> is the relative path to and name of the CloudFormation template YAML file that you saved.

   3

   <parameters> is the relative path to and name of the CloudFormation parameters JSON file.

5. Confirm that the template components exist:

   $ aws cloudformation describe-stacks --stack-name <name>

Procedure

1. Change to the directory that contains the installation program and run the following command:

   $ ./openshift-install wait-for bootstrap-complete --dir=<installation_directory> \ 1
       --log-level info 2

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

   2

   To view different installation details, specify warn, debug, or error instead of info.

   If the command exits without a FATAL warning, your production control plane has initialized.

Procedure

1. From the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site, navigate to the page for your installation type and click Download Command-line Tools.

2. From the site that is displayed, download the compressed file for your operating system.

   Note

   You can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Procedure

1. Export the kubeadmin credentials:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig 1
   
   $ oc whoami
   system:admin

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes
   
   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.13.4+b626c2fe1
   master-1  Ready     master  63m  v1.13.4+b626c2fe1
   master-2  Ready     master  64m  v1.13.4+b626c2fe1
   worker-0  NotReady  worker  76s  v1.13.4+b626c2fe1
   worker-1  NotReady  worker  70s  v1.13.4+b626c2fe1

   The output lists all of the machines that you created.

2. Review the pending certificate signing requests (CSRs) and ensure that the you see a client and server request with Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr
   
   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 1
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending 2
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

   1

   A client request CSR.

   2

   A server request CSR.

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. You must implement a method of automatically approving the kubelet serving certificate requests.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • If all the CSRs are valid, approve them all by running the following command:

     $ oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

Procedure

1. Watch the cluster components come online:

   $ watch -n5 oc get clusteroperators
   
   NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                       4.1.0     True        False         False      69s
   cloud-credential                     4.1.0     True        False         False      12m
   cluster-autoscaler                   4.1.0     True        False         False      11m
   console                              4.1.0     True        False         False      46s
   dns                                  4.1.0     True        False         False      11m
   image-registry                       4.1.0     False       True          False      5m26s
   ingress                              4.1.0     True        False         False      5m36s
   kube-apiserver                       4.1.0     True        False         False      8m53s
   kube-controller-manager              4.1.0     True        False         False      7m24s
   kube-scheduler                       4.1.0     True        False         False      12m
   machine-api                          4.1.0     True        False         False      12m
   machine-config                       4.1.0     True        False         False      7m36s
   marketplace                          4.1.0     True        False         False      7m54m
   monitoring                           4.1.0     True        False         False      7h54s
   network                              4.1.0     True        False         False      5m9s
   node-tuning                          4.1.0     True        False         False      11m
   openshift-apiserver                  4.1.0     True        False         False      11m
   openshift-controller-manager         4.1.0     True        False         False      5m943s
   openshift-samples                    4.1.0     True        False         False      3m55s
   operator-lifecycle-manager           4.1.0     True        False         False      11m
   operator-lifecycle-manager-catalog   4.1.0     True        False         False      11m
   service-ca                           4.1.0     True        False         False      11m
   service-catalog-apiserver            4.1.0     True        False         False      5m26s
   service-catalog-controller-manager   4.1.0     True        False         False      5m25s
   storage                              4.1.0     True        False         False      5m30s

2. Configure the Operators that are not available.

Procedure

1. Delete the bootstrap resources. If you used the CloudFormation template, delete its stack:

   $ aws cloudformation delete-stack --stack-name <name> 1

   1

   <name> is the name of your bootstrap stack.

2. Complete the cluster installation:

   $ ./openshift-install --dir=<installation_directory> wait-for install-complete 1
   
   INFO Waiting up to 30m0s for the cluster to initialize...

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

   Important

   The Ignition config files that the installation program generates contain certificates that expire after 24 hours. You must keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

Procedure

1. Configure DHCP.
2. Provision the required load balancers.
3. Configure the ports for your machines.
4. Configure DNS.
5. Ensure network connectivity.

Procedure

1. If you do not have an SSH key that is configured for password-less authentication on your computer, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t rsa -b 4096 -N '' \
       -f <path>/<file_name> 1

   1

   Specify the path and file name, such as ~/.ssh/id_rsa, of the SSH key.

   Running this command generates an SSH key that does not require a password in the location that you specified.

2. Start the ssh-agent process as a background task:

   $ eval "$(ssh-agent -s)"
   
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> 1
   
   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   1

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_rsa

Procedure

1. Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

2. Navigate to the page for your installation type, download the installation program for your operating system, and place the file in the directory where you will store the installation configuration files.

   Important

   The installation program creates several files on the computer that you use to install your cluster. You must keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar xvf <installation_program>.tar.gz

4. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret as a .txt file or copy it to your clipboard. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

Procedure

1. From the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site, navigate to the page for your installation type and click Download Command-line Tools.

2. From the site that is displayed, download the compressed file for your operating system.

   Note

   You can install oc on Linux, Windows, or macOS.

3. Extract the compressed file and place it in a directory that is on your PATH.

Procedure

1. Create an installation directory to store your required installation assets in:

   $ mkdir <installation_directory>

   Important

   You must create a directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

2. Customize the following install-config.yaml file template and save it in the <installation_directory>.

   Note

   You must name this configuration file install-config.yaml.

3. Back up the install-config.yaml file so that you can use it to install multiple clusters.

   Important

   The install-config.yaml file is consumed during the next step of the installation process. You must back it up now.

Procedure

1. Obtain the Ignition config files:

   $ ./openshift-install create ignition-configs --dir=<installation_directory> 1

   1

   For <installation_directory>, specify the directory name to store the files that the installation program creates.

   Important

   If you created an install-config.yaml file, specify the directory that contains it. Otherwise, specify an empty directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

   The following files are generated in the directory:

   .
   ├── auth
   │   ├── kubeadmin-password
   │   └── kubeconfig
   ├── bootstrap.ign
   ├── master.ign
   ├── metadata.json
   └── worker.ign

Procedure

1. Monitor the bootstrap process:

   $ ./openshift-install --dir=<installation_directory> wait-for bootstrap-complete \ 1
       --log-level info 2
   
   INFO Waiting up to 30m0s for the Kubernetes API at https://api.test.example.com:6443...
   INFO API v1.13.4+b626c2fe1 up
   INFO Waiting up to 30m0s for the bootstrap-complete event...

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

   2

   To view different installation details, specify warn, debug, or error instead of info.

   The command succeeds when the Kubernetes API server signals that it has been bootstrapped on the control plane machines.

2. After bootstrap process is complete, remove the bootstrap machine from the load balancer.

   Important

   You must remove the bootstrap machine from the load balancer at this point. You can also remove or reformat the machine itself.

Procedure

1. Export the kubeadmin credentials:

   $ export KUBECONFIG=<installation_directory>/auth/kubeconfig 1
   
   $ oc whoami
   system:admin

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

Procedure

1. Confirm that the cluster recognizes the machines:

   $ oc get nodes
   
   NAME      STATUS    ROLES   AGE  VERSION
   master-0  Ready     master  63m  v1.13.4+b626c2fe1
   master-1  Ready     master  63m  v1.13.4+b626c2fe1
   master-2  Ready     master  64m  v1.13.4+b626c2fe1
   worker-0  NotReady  worker  76s  v1.13.4+b626c2fe1
   worker-1  NotReady  worker  70s  v1.13.4+b626c2fe1

   The output lists all of the machines that you created.

2. Review the pending certificate signing requests (CSRs) and ensure that the you see a client and server request with Pending or Approved status for each machine that you added to the cluster:

   $ oc get csr
   
   NAME        AGE     REQUESTOR                                                                   CONDITION
   csr-8b2br   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending 1
   csr-8vnps   15m     system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-bfd72   5m26s   system:node:ip-10-0-50-126.us-east-2.compute.internal                       Pending 2
   csr-c57lv   5m26s   system:node:ip-10-0-95-157.us-east-2.compute.internal                       Pending
   ...

   1

   A client request CSR.

   2

   A server request CSR.

   In this example, two machines are joining the cluster. You might see more approved CSRs in the list.

3. If the CSRs were not approved, after all of the pending CSRs for the machines you added are in Pending status, approve the CSRs for your cluster machines:

   Note

   Because the CSRs rotate automatically, approve your CSRs within an hour of adding the machines to the cluster. If you do not approve them within an hour, the certificates will rotate, and more than two certificates will be present for each node. You must approve all of these certificates. After you approve the initial CSRs, the subsequent node client CSRs are automatically approved by the cluster kube-controller-manager. You must implement a method of automatically approving the kubelet serving certificate requests.

   • To approve them individually, run the following command for each valid CSR:

     $ oc adm certificate approve <csr_name> 1

     1

     <csr_name> is the name of a CSR from the list of current CSRs.

   • If all the CSRs are valid, approve them all by running the following command:

     $ oc get csr -ojson | jq -r '.items[] | select(.status == {} ) | .metadata.name' | xargs oc adm certificate approve

Procedure

1. Watch the cluster components come online:

   $ watch -n5 oc get clusteroperators
   
   NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                       4.1.0     True        False         False      69s
   cloud-credential                     4.1.0     True        False         False      12m
   cluster-autoscaler                   4.1.0     True        False         False      11m
   console                              4.1.0     True        False         False      46s
   dns                                  4.1.0     True        False         False      11m
   image-registry                       4.1.0     False       True          False      5m26s
   ingress                              4.1.0     True        False         False      5m36s
   kube-apiserver                       4.1.0     True        False         False      8m53s
   kube-controller-manager              4.1.0     True        False         False      7m24s
   kube-scheduler                       4.1.0     True        False         False      12m
   machine-api                          4.1.0     True        False         False      12m
   machine-config                       4.1.0     True        False         False      7m36s
   marketplace                          4.1.0     True        False         False      7m54m
   monitoring                           4.1.0     True        False         False      7h54s
   network                              4.1.0     True        False         False      5m9s
   node-tuning                          4.1.0     True        False         False      11m
   openshift-apiserver                  4.1.0     True        False         False      11m
   openshift-controller-manager         4.1.0     True        False         False      5m943s
   openshift-samples                    4.1.0     True        False         False      3m55s
   operator-lifecycle-manager           4.1.0     True        False         False      11m
   operator-lifecycle-manager-catalog   4.1.0     True        False         False      11m
   service-ca                           4.1.0     True        False         False      11m
   service-catalog-apiserver            4.1.0     True        False         False      5m26s
   service-catalog-controller-manager   4.1.0     True        False         False      5m25s
   storage                              4.1.0     True        False         False      5m30s

2. Configure the Operators that are not available.

Procedure

1. Confirm that all the cluster components are online:

   $ watch -n5 oc get clusteroperators
   
   NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                       4.1.0     True        False         False      10m
   cloud-credential                     4.1.0     True        False         False      22m
   cluster-autoscaler                   4.1.0     True        False         False      21m
   console                              4.1.0     True        False         False      10m
   dns                                  4.1.0     True        False         False      21m
   image-registry                       4.1.0     True        False         False      16m
   ingress                              4.1.0     True        False         False      16m
   kube-apiserver                       4.1.0     True        False         False      19m
   kube-controller-manager              4.1.0     True        False         False      18m
   kube-scheduler                       4.1.0     True        False         False      22m
   machine-api                          4.1.0     True        False         False      22m
   machine-config                       4.1.0     True        False         False      18m
   marketplace                          4.1.0     True        False         False      18m
   monitoring                           4.1.0     True        False         False      18m
   network                              4.1.0     True        False         False      16m
   node-tuning                          4.1.0     True        False         False      21m
   openshift-apiserver                  4.1.0     True        False         False      21m
   openshift-controller-manager         4.1.0     True        False         False      17m
   openshift-samples                    4.1.0     True        False         False      14m
   operator-lifecycle-manager           4.1.0     True        False         False      21m
   operator-lifecycle-manager-catalog   4.1.0     True        False         False      21m
   service-ca                           4.1.0     True        False         False      21m
   service-catalog-apiserver            4.1.0     True        False         False      16m
   service-catalog-controller-manager   4.1.0     True        False         False      16m
   storage                              4.1.0     True        False         False      16m

   When all of the cluster Operators are AVAILABLE, you can complete the installation.

2. Monitor for cluster completion:

   $ ./openshift-install --dir=<installation_directory> wait-for install-complete 1
   INFO Waiting up to 30m0s for the cluster to initialize...

   1

   For <installation_directory>, specify the path to the directory that you stored the installation files in.

   The command succeeds when the Cluster Version Operator finishes deploying the OpenShift Container Platform cluster from Kubernetes API server.

   Important

   The Ignition config files that the installation program generates contain certificates that expire after 24 hours. You must keep the cluster running for 24 hours in a non-degraded state to ensure that the first certificate rotation has finished.

3. Confirm that the Kubernetes API server is communicating with the Pods.

   1. To view a list of all Pods, use the following command:

      $ oc get pods --all-namespaces
      
      NAMESPACE                         NAME                                            READY   STATUS      RESTARTS   AGE
      openshift-apiserver-operator      openshift-apiserver-operator-85cb746d55-zqhs8   1/1     Running     1          9m
      openshift-apiserver               apiserver-67b9g                                 1/1     Running     0          3m
      openshift-apiserver               apiserver-ljcmx                                 1/1     Running     0          1m
      openshift-apiserver               apiserver-z25h4                                 1/1     Running     0          2m
      openshift-authentication-operator authentication-operator-69d5d8bf84-vh2n8        1/1     Running     0          5m
      ...

   2. View the logs for a Pod that is listed in the output of the previous command by using the following command:

      $ oc logs <pod_name> -n <namespace> 1

      1

      Specify the Pod name and namespace, as shown in the output of the previous command.

      If the Pod logs display, the Kubernetes API server can communicate with the cluster machines.

Procedure

1. Configure DHCP.
2. Provision the required load balancers.
3. Configure the ports for your machines.
4. Configure DNS.
5. Ensure network connectivity.

Procedure

1. If you do not have an SSH key that is configured for password-less authentication on your computer, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t rsa -b 4096 -N '' \
       -f <path>/<file_name> 1

   1

   Specify the path and file name, such as ~/.ssh/id_rsa, of the SSH key.

   Running this command generates an SSH key that does not require a password in the location that you specified.

2. Start the ssh-agent process as a background task:

   $ eval "$(ssh-agent -s)"
   
   Agent pid 31874

3. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> 1
   
   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

   1

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_rsa

Procedure

1. Access the Infrastructure Provider page on the Red Hat OpenShift Cluster Manager site. If you have a Red Hat account, log in with your credentials. If you do not, create an account.

2. Navigate to the page for your installation type, download the installation program for your operating system, and place the file in the directory where you will store the installation configuration files.

   Important

   The installation program creates several files on the computer that you use to install your cluster. You must keep both the installation program and the files that the installation program creates after you finish installing the cluster.

3. Extract the installation program. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar xvf <installation_program>.tar.gz

4. From the Pull Secret page on the Red Hat OpenShift Cluster Manager site, download your installation pull secret as a .txt file or copy it to your clipboard. This pull secret allows you to authenticate with the services that are provided by the included authorities, including Quay.io, which serves the container images for OpenShift Container Platform components.

Procedure

1. Create an installation directory to store your required installation assets in:

   $ mkdir <installation_directory>

   Important

   You must create a directory. Some installation assets, like bootstrap X.509 certificates have short expiration intervals, so you must not reuse an installation directory. If you want to reuse individual files from another cluster installation, you can copy them into your directory. However, the file names for the installation assets might change between releases. Use caution when copying installation files from an earlier OpenShift Container Platform version.

2. Customize the following install-config.yaml file template and save it in the <installation_directory>.

   Note

   You must name this configuration file install-config.yaml.

3. Back up the install-config.yaml file so that you can use it to install multiple clusters.

   Important

   The install-config.yaml file is consumed during the next step of the installation process. You must back it up now.