Chapter 5. Installing a cluster on RHV in a restricted network

Procedure

1. Check that the RHV version supports installation of OpenShift Container Platform version 4.12.

   1. In the RHV Administration Portal, click the ? help icon in the upper-right corner and select About.
   2. In the window that opens, make a note of the RHV Software Version.
   3. Confirm that the RHV version is 4.4. For more information about supported version combinations, see Support Matrix for OpenShift Container Platform on RHV.

2. Inspect the data center, cluster, and storage.

   1. In the RHV Administration Portal, click Compute Data Centers.
   2. Confirm that the data center where you plan to install OpenShift Container Platform is accessible.
   3. Click the name of that data center.
   4. In the data center details, on the Storage tab, confirm the storage domain where you plan to install OpenShift Container Platform is Active.
   5. Record the Domain Name for use later on.
   6. Confirm Free Space has at least 230 GiB.
   7. Confirm that the storage domain meets these etcd backend performance requirements, which you can measure by using the fio performance benchmarking tool.
   8. In the data center details, click the Clusters tab.
   9. Find the RHV cluster where you plan to install OpenShift Container Platform. Record the cluster name for use later on.

3. Inspect the RHV host resources.

   1. In the RHV Administration Portal, click Compute > Clusters.
   2. Click the cluster where you plan to install OpenShift Container Platform.
   3. In the cluster details, click the Hosts tab.
   4. Inspect the hosts and confirm they have a combined total of at least 28 Logical CPU Cores available exclusively for the OpenShift Container Platform cluster.
   5. Record the number of available Logical CPU Cores for use later on.
   6. Confirm that these CPU cores are distributed so that each of the seven virtual machines created during installation can have four cores.

   7. Confirm that, all together, the hosts have 112 GiB of Max free Memory for scheduling new virtual machines distributed to meet the requirements for each of the following OpenShift Container Platform machines:

      • 16 GiB required for the bootstrap machine
      • 16 GiB required for each of the three control plane machines
      • 16 GiB for each of the three compute machines
   8. Record the amount of Max free Memory for scheduling new virtual machines for use later on.

4. Verify that the virtual network for installing OpenShift Container Platform has access to the RHV Manager’s REST API. From a virtual machine on this network, use curl to reach the RHV Manager’s REST API:

   $ curl -k -u <username>@<profile>:<password> \ 1
   https://<engine-fqdn>/ovirt-engine/api 2

   1

   For <username>, specify the user name of an RHV account with privileges to create and manage an OpenShift Container Platform cluster on RHV. For <profile>, specify the login profile, which you can get by going to the RHV Administration Portal login page and reviewing the Profile dropdown list. For <password>, specify the password for that user name.

   2

   For <engine-fqdn>, specify the fully qualified domain name of the RHV environment.

   For example:

   $ curl -k -u ocpadmin@internal:pw123 \
   https://rhv-env.virtlab.example.com/ovirt-engine/api

Procedure

1. Update or install Python3 and Ansible. For example:

   # dnf update python3 ansible

2. Install the python3-ovirt-engine-sdk4 package to get the Python Software Development Kit.

3. Install the ovirt.image-template Ansible role. On the RHV Manager and other Red Hat Enterprise Linux (RHEL) machines, this role is distributed as the ovirt-ansible-image-template package. For example, enter:

   # dnf install ovirt-ansible-image-template

4. Install the ovirt.vm-infra Ansible role. On the RHV Manager and other RHEL machines, this role is distributed as the ovirt-ansible-vm-infra package.

   # dnf install ovirt-ansible-vm-infra

5. Create an environment variable and assign an absolute or relative path to it. For example, enter:

   $ export ASSETS_DIR=./wrk

   Note

   The installation program uses this variable to create a directory where it saves important installation-related files. Later, the installation process reuses this variable to locate those asset files. Avoid deleting this assets directory; it is required for uninstalling the cluster.

Procedure

1. Use either of these two methods to download the CA certificate:

   • Go to the Manager’s webpage, https://<engine-fqdn>/ovirt-engine/. Then, under Downloads, click the CA Certificate link.

   • Run the following command:

     $ curl -k 'https://<engine-fqdn>/ovirt-engine/services/pki-resource?resource=ca-certificate&format=X509-PEM-CA' -o /tmp/ca.pem  1

     1

     For <engine-fqdn>, specify the fully qualified domain name of the RHV Manager, such as rhv-env.virtlab.example.com.

2. Configure the CA file to grant rootless user access to the Manager. Set the CA file permissions to have an octal value of 0644 (symbolic value: -rw-r—​r--):

   $ sudo chmod 0644 /tmp/ca.pem

3. For Linux, copy the CA certificate to the directory for server certificates. Use -p to preserve the permissions:

   $ sudo cp -p /tmp/ca.pem /etc/pki/ca-trust/source/anchors/ca.pem

4. Add the certificate to the certificate manager for your operating system:

   • For macOS, double-click the certificate file and use the Keychain Access utility to add the file to the System keychain.

   • For Linux, update the CA trust:

     $ sudo update-ca-trust

     Note

     If you use your own certificate authority, make sure the system trusts it.

Procedure

1. If you do not have an existing SSH key pair on your local machine to use for authentication onto your cluster nodes, create one. For example, on a computer that uses a Linux operating system, run the following command:

   $ ssh-keygen -t ed25519 -N '' -f <path>/<file_name> 1

   1

   Specify the path and file name, such as ~/.ssh/id_ed25519, of the new SSH key. If you have an existing key pair, ensure your public key is in the your ~/.ssh directory.

   Note

   If you plan to install an OpenShift Container Platform cluster that uses FIPS validated or Modules In Process cryptographic libraries on the x86_64, ppc64le, and s390x architectures. do not create a key that uses the ed25519 algorithm. Instead, create a key that uses the rsa or ecdsa algorithm.

2. View the public SSH key:

   $ cat <path>/<file_name>.pub

   For example, run the following to view the ~/.ssh/id_ed25519.pub public key:

   $ cat ~/.ssh/id_ed25519.pub

3. Add the SSH private key identity to the SSH agent for your local user, if it has not already been added. SSH agent management of the key is required for password-less SSH authentication onto your cluster nodes, or if you want to use the ./openshift-install gather command.

   Note

   On some distributions, default SSH private key identities such as ~/.ssh/id_rsa and ~/.ssh/id_dsa are managed automatically.

   1. If the ssh-agent process is not already running for your local user, start it as a background task:

      $ eval "$(ssh-agent -s)"

      Example output

      Agent pid 31874

      Note

      If your cluster is in FIPS mode, only use FIPS-compliant algorithms to generate the SSH key. The key must be either RSA or ECDSA.

4. Add your SSH private key to the ssh-agent:

   $ ssh-add <path>/<file_name> 1

   1

   Specify the path and file name for your SSH private key, such as ~/.ssh/id_ed25519

   Example output

   Identity added: /home/<you>/<path>/<file_name> (<computer_name>)

Procedure

1. Locate the RHCOS image download page for the version of OpenShift Container Platform you are installing, such as Index of /pub/openshift-v4/dependencies/rhcos/latest/latest.
2. From that download page, copy the URL of an OpenStack qcow2 image, such as https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.12/latest/rhcos-openstack.x86_64.qcow2.gz.

3. Edit the inventory.yml playbook you downloaded earlier. In it, paste the URL as the value for image_url. For example:

   rhcos:
     "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.12/latest/rhcos-openstack.x86_64.qcow2.gz"

Procedure

1. Run the installation program:

   $ openshift-install create install-config --dir $ASSETS_DIR

2. Respond to the installation program’s prompts with information about your system.

   Example output

   ? SSH Public Key /home/user/.ssh/id_dsa.pub
   ? Platform <ovirt>
   ? Engine FQDN[:PORT] [? for help] <engine.fqdn>
   ? Enter ovirt-engine username <ocpadmin@internal>
   ? Enter password <******>
   ? oVirt cluster <cluster>
   ? oVirt storage <storage>
   ? oVirt network <net>
   ? Internal API virtual IP <172.16.0.252>
   ? Ingress virtual IP <172.16.0.251>
   ? Base Domain <example.org>
   ? Cluster Name <ocp4>
   ? Pull Secret [? for help] <********>

Procedure

1. Set the number of compute nodes to zero replicas:

   $ python3 -c 'import os, yaml
   path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"]
   conf = yaml.safe_load(open(path))
   conf["compute"][0]["replicas"] = 0
   open(path, "w").write(yaml.dump(conf, default_flow_style=False))'

2. Set the IP range of the machine network. For example, to set the range to 172.16.0.0/16, enter:

   $ python3 -c 'import os, yaml
   path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"]
   conf = yaml.safe_load(open(path))
   conf["networking"]["machineNetwork"][0]["cidr"] = "172.16.0.0/16"
   open(path, "w").write(yaml.dump(conf, default_flow_style=False))'

3. Remove the ovirt section and change the platform to none:

   $ python3 -c 'import os, yaml
   path = "%s/install-config.yaml" % os.environ["ASSETS_DIR"]
   conf = yaml.safe_load(open(path))
   platform = conf["platform"]
   del platform["ovirt"]
   platform["none"] = {}
   open(path, "w").write(yaml.dump(conf, default_flow_style=False))'

   Warning

   Red Hat Virtualization does not currently support installation with user-provisioned infrastructure on the oVirt platform. Therefore, you must set the platform to none, allowing OpenShift Container Platform to identify each node as a bare-metal node and the cluster as a bare-metal cluster. This is the same as installing a cluster on any platform, and has the following limitations:

   1. There will be no cluster provider so you must manually add each machine and there will be no node scaling capabilities.
   2. The oVirt CSI driver will not be installed and there will be no CSI capabilities.

Procedure

1. Optional: Create a backup copy of the install-config.yaml file:

   $ cp install-config.yaml install-config.yaml.backup

2. Generate a set of manifests in your assets directory:

   $ openshift-install create manifests --dir $ASSETS_DIR

   This command displays the following messages.

   Example output

   INFO Consuming Install Config from target directory
   WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings

   The command generates the following manifest files:

   Example output

   $ tree
   .
   └── wrk
       ├── manifests
       │   ├── 04-openshift-machine-config-operator.yaml
       │   ├── cluster-config.yaml
       │   ├── cluster-dns-02-config.yml
       │   ├── cluster-infrastructure-02-config.yml
       │   ├── cluster-ingress-02-config.yml
       │   ├── cluster-network-01-crd.yml
       │   ├── cluster-network-02-config.yml
       │   ├── cluster-proxy-01-config.yaml
       │   ├── cluster-scheduler-02-config.yml
       │   ├── cvo-overrides.yaml
       │   ├── etcd-ca-bundle-configmap.yaml
       │   ├── etcd-client-secret.yaml
       │   ├── etcd-host-service-endpoints.yaml
       │   ├── etcd-host-service.yaml
       │   ├── etcd-metric-client-secret.yaml
       │   ├── etcd-metric-serving-ca-configmap.yaml
       │   ├── etcd-metric-signer-secret.yaml
       │   ├── etcd-namespace.yaml
       │   ├── etcd-service.yaml
       │   ├── etcd-serving-ca-configmap.yaml
       │   ├── etcd-signer-secret.yaml
       │   ├── kube-cloud-config.yaml
       │   ├── kube-system-configmap-root-ca.yaml
       │   ├── machine-config-server-tls-secret.yaml
       │   └── openshift-config-secret-pull-secret.yaml
       └── openshift
           ├── 99_kubeadmin-password-secret.yaml
           ├── 99_openshift-cluster-api_master-user-data-secret.yaml
           ├── 99_openshift-cluster-api_worker-user-data-secret.yaml
           ├── 99_openshift-machineconfig_99-master-ssh.yaml
           ├── 99_openshift-machineconfig_99-worker-ssh.yaml
           └── openshift-install-manifests.yaml

Procedure

1. To make the control plane nodes non-schedulable, enter:

   $ python3 -c 'import os, yaml
   path = "%s/manifests/cluster-scheduler-02-config.yml" % os.environ["ASSETS_DIR"]
   data = yaml.safe_load(open(path))
   data["spec"]["mastersSchedulable"] = False
   open(path, "w").write(yaml.dump(data, default_flow_style=False))'

Procedure

1. To build the Ignition files, enter:

   $ openshift-install create ignition-configs --dir $ASSETS_DIR

   Example output

   $ tree
   .
   └── wrk
       ├── auth
       │   ├── kubeadmin-password
       │   └── kubeconfig
       ├── bootstrap.ign
       ├── master.ign
       ├── metadata.json
       └── worker.ign

Procedure

1. In inventory.yml, under the control_plane and compute variables, change both instances of type: high_performance to type: server.

2. Optional: If you plan to perform multiple installations to the same cluster, create different templates for each OpenShift Container Platform installation. In the inventory.yml file, prepend the value of template with infraID. For example:

     control_plane:
       cluster: "{{ ovirt_cluster }}"
       memory: 16GiB
       sockets: 4
       cores: 1
       template: "{{ metadata.infraID }}-rhcos_tpl"
       operating_system: "rhcos_x64"
       ...

3. Create the templates and virtual machines:

   $ ansible-playbook -i inventory.yml create-templates-and-vms.yml

Procedure

1. Create the bootstrap machine:

   $ ansible-playbook -i inventory.yml bootstrap.yml

2. Connect to the bootstrap machine using a console in the Administration Portal or SSH. Replace <bootstrap_ip> with the bootstrap node IP address. To use SSH, enter:

   $ ssh core@<boostrap.ip>

3. Collect bootkube.service journald unit logs for the release image service from the bootstrap node:

   [core@ocp4-lk6b4-bootstrap ~]$ journalctl -b -f -u release-image.service -u bootkube.service

   Note

   The bootkube.service log on the bootstrap node outputs etcd connection refused errors, indicating that the bootstrap server is unable to connect to etcd on control plane nodes. After etcd has started on each control plane node and the nodes have joined the cluster, the errors should stop.

Procedure

1. Create the control plane nodes:

   $ ansible-playbook -i inventory.yml masters.yml

2. While the playbook creates your control plane, monitor the bootstrapping process:

   $ openshift-install wait-for bootstrap-complete --dir $ASSETS_DIR

   Example output

   INFO API v1.25.0 up
   INFO Waiting up to 40m0s for bootstrapping to complete...

3. When all the pods on the control plane nodes and etcd are up and running, the installation program displays the following output.

   Example output

   INFO It is now safe to remove the bootstrap resources

Procedure

1. In the cluster environment, export the administrator’s kubeconfig file:

   $ export KUBECONFIG=$ASSETS_DIR/auth/kubeconfig

   The kubeconfig file contains information about the cluster that is used by the CLI to connect a client to the correct cluster and API server.

2. View the control plane and compute machines created after a deployment:

   $ oc get nodes

3. View your cluster’s version:

   $ oc get clusterversion

4. View your Operators' status:

   $ oc get clusteroperator

5. View all running pods in the cluster:

   $ oc get pods -A

Procedure

1. To remove the bootstrap machine from the cluster, enter:

   $ ansible-playbook -i inventory.yml retire-bootstrap.yml

2. Remove settings for the bootstrap machine from the load balancer directives.

Procedure

1. Create the worker nodes:

   $ ansible-playbook -i inventory.yml workers.yml

2. To list all of the CSRs, enter:

   $ oc get csr -A

   Eventually, this command displays one CSR per node. For example:

   Example output

   NAME        AGE    SIGNERNAME                                    REQUESTOR                                                                   CONDITION
   csr-2lnxd   63m    kubernetes.io/kubelet-serving                 system:node:ocp4-lk6b4-master0.ocp4.example.org                             Approved,Issued
   csr-hff4q   64m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
   csr-hsn96   60m    kubernetes.io/kubelet-serving                 system:node:ocp4-lk6b4-master2.ocp4.example.org                             Approved,Issued
   csr-m724n   6m2s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-p4dz2   60m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
   csr-t9vfj   60m    kubernetes.io/kubelet-serving                 system:node:ocp4-lk6b4-master1.ocp4.example.org                             Approved,Issued
   csr-tggtr   61m    kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
   csr-wcbrf   7m6s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending

3. To filter the list and see only pending CSRs, enter:

   $ watch "oc get csr -A | grep pending -i"

   This command refreshes the output every two seconds and displays only pending CSRs. For example:

   Example output

   Every 2.0s: oc get csr -A | grep pending -i
   
   csr-m724n   10m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending
   csr-wcbrf   11m   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Pending

4. Inspect each pending request. For example:

   Example output

   $ oc describe csr csr-m724n

   Example output

   Name:               csr-m724n
   Labels:             <none>
   Annotations:        <none>
   CreationTimestamp:  Sun, 19 Jul 2020 15:59:37 +0200
   Requesting User:    system:serviceaccount:openshift-machine-config-operator:node-bootstrapper
   Signer:             kubernetes.io/kube-apiserver-client-kubelet
   Status:             Pending
   Subject:
            Common Name:    system:node:ocp4-lk6b4-worker1.ocp4.example.org
            Serial Number:
            Organization:   system:nodes
   Events:  <none>

5. If the CSR information is correct, approve the request:

   $ oc adm certificate approve csr-m724n

6. Wait for the installation process to finish:

   $ openshift-install wait-for install-complete --dir $ASSETS_DIR --log-level debug

   When the installation completes, the command line displays the URL of the OpenShift Container Platform web console and the administrator user name and password.