Assisted Installer for OpenShift Container Platform

Procedure

1. Log in to the Red Hat Hybrid Cloud Console.
2. In the Red Hat OpenShift tile, click Scale your applications.
3. In the menu, click Clusters.
4. Click Create cluster.
5. Click the Datacenter tab.
6. Under Assisted Installer, click Create cluster.
7. Enter a name for the cluster in the Cluster name field.

8. Enter a base domain for the cluster in the Base domain field. All subdomains for the cluster will use this base domain.

   Note

   The base domain must be a valid DNS name. You must not have a wild card domain set up for the base domain.

9. Select the version of OpenShift Container Platform to install.

   Important
   • For IBM Power and IBM zSystems platforms, only OpenShift Container Platform version 4.13 and later is supported.
   • For a mixed-architecture cluster installation, select OpenShift Container Platform version 4.12 or later, and use the -multi option. For instructions on installing a mixed-architecture cluster, see Additional resources.

10. Optional: Select Install single node Openshift (SNO) if you want to install OpenShift Container Platform on a single node.

    Note

    Currently, SNO is not supported on IBM zSystems and IBM Power platforms.

11. Optional: The Assisted Installer already has the pull secret associated to your account. If you want to use a different pull secret, select Edit pull secret.

12. Optional: If you are installing OpenShift Container Platform on a third-party platform, select the platform from the Integrate with external parter platforms list. Valid values are Nutanix, vSphere or Oracle Cloud Infrastructure. Assisted Installer defaults to having no platform integration.

    Note

    For details on each of the external partner integrations, see Additional Resources.

    Important

    Assisted Installer supports Oracle Cloud Infrastructure (OCI) integration from OpenShift Container Platform 4.14 and later. For OpenShift Container Platform 4.14, the OCI integration is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

    For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features - Scope of Support.

13. Optional: Assisted Installer defaults to using x86_64 CPU architecture. If you are installing OpenShift Container Platform on a different architecture select the respective architecture to use. Valid values are arm64, ppc64le, and s390x. Keep in mind, some features are not available with arm64, ppc64le, and s390x CPU architectures.

    Important

    For a mixed-architecture cluster installation, use the default x86_64 architecture. For instructions on installing a mixed-architecture cluster, see Additional resources.

14. Optional: The Assisted Installer defaults to DHCP networking. If you are using a static IP configuration, bridges or bonds for the cluster nodes instead of DHCP reservations, select Static IP, bridges, and bonds.

    Note

    A Static IP configuration is not supported for OpenShift Container Platform installations on Oracle Cloud Infrastructure (OCI).

15. Optional: If you want to enable encryption of the installation disks, under Enable encryption of installation disks you can select Control plane node, worker for single-node OpenShift. For multi-node clusters, you can select Control plane nodes to encrypt the control plane node installation disks and select Workers to encrypt worker node installation disks.

Procedure

1. Select the internet protocol version. Valid options are IPv4 and Dual stack.
2. If the cluster hosts are on a shared VLAN, enter the VLAN ID.

3. Enter the network-wide IP addresses. If you selected Dual stack networking, you must enter both IPv4 and IPv6 addresses.

   1. Enter the cluster network’s IP address range in CIDR notation.
   2. Enter the default gateway IP address.
   3. Enter the DNS server IP address.

4. Enter the host-specific configuration.

   1. If you are only setting a static IP address that uses a single network interface, use the form view to enter the IP address and the MAC address for each host.
   2. If you use multiple interfaces, bonding, or other advanced networking features, use the YAML view and enter the desired network state for each host that uses NMState syntax. Then, add the MAC address and interface name for each host interface used in your network configuration.

Procedure

1. To install OpenShift Virtualization, select Install OpenShift Virtualization.
2. To install Multicluster Engine (MCE), select Install multicluster engine.
3. To install OpenShift Data Foundation, select Install OpenShift Data Foundation.
4. To install Logical Volume Manager, select Install Logical Volume Manager.
5. Click Next to proceed to the next step.

Procedure

1. Click the Add hosts button and select the provisioning type.

   1. Select Minimal image file: Provision with virtual media to download a smaller image that will fetch the data needed to boot. The nodes must have virtual media capability. This is the recommended method for x86_64 and arm64 architectures.
   2. Select Full image file: Provision with physical media to download the larger full image. This is the recommended method for the ppc64le architecture and for the s390x architecture when installing with RHEL KVM.

   3. Select iPXE: Provision from your network server to boot the hosts using iPXE.

      Note
      • If you install on RHEL KVM, in some circumstances, the VMs on the KVM host are not rebooted on first boot and need to be restarted manually.
      • If you install OpenShift Container Platform on Oracle Cloud Infrastructure, select Minimal image file: Provision with virtual media only.
2. Optional: If the cluster hosts are behind a firewall that requires the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, IP address and port for the HTTP and HTTPS URLs of the proxy server.

3. Optional: Add an SSH public key so that you can connect to the cluster nodes as the core user. Having a login to the cluster nodes can provide you with debugging information during the installation.

   Important

   Do not skip this procedure in production environments, where disaster recovery and debugging is required.

   1. If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
   2. In the SSH public key field, click Browse to upload the id_rsa.pub file containing the SSH public key. Alternatively, drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
4. Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy, or if the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates. Add additional certificates in X.509 format.
5. Configure the discovery image if needed.
6. Optional: If you are installing on a platform and want to integrate with the platform, select Integrate with your virtualization platform. You must boot all hosts and ensure they appear in the host inventory. All the hosts must be on the same platform.
7. Click Generate Discovery ISO or Generate Script File.
8. Download the discovery ISO or iPXE script.
9. Boot the host(s) with the discovery image or iPXE script.

Procedure

1. From the Options (⋮) menu for a host, select Change hostname. If necessary, enter a new name for the host and click Change. You must ensure that each host has a valid and unique hostname.

   Alternatively, from the Actions list, select Change hostname to rename multiple selected hosts. In the Change Hostname dialog, type the new name and include {{n}} to make each hostname unique. Then click Change.

   Note

   You can see the new names appearing in the Preview pane as you type. The name will be identical for all selected hosts, with the exception of a single-digit increment per host.

2. From the Options (⋮) menu, you can select Delete host to delete a host. Click Delete to confirm the deletion.

   Alternatively, from the Actions list, select Delete to delete multiple selected hosts at the same time. Then click Delete hosts.

   Note

   In a regular deployment, a cluster can have three or more hosts, and three of these must be control plane hosts. If you delete a host that is also a control plane, or if you are left with only two hosts, you will get a message saying that the system is not ready. To restore a host, you will need to reboot it from the discovery ISO.

3. From the Options (⋮) menu for the host, optionally select View host events. The events in the list are presented chronologically.

4. For multi-host clusters, in the Role column next to the host name, you can click on the menu to change the role of the host.

   If you do not select a role, the Assisted Installer will assign the role automatically. The minimum hardware requirements for control plane nodes exceed that of worker nodes. If you assign a role to a host, ensure that you assign the control plane role to hosts that meet the minimum hardware requirements.

5. Click the Status link to view hardware, network and operator validations for the host.
6. Click the arrow to the left of a host name to expand the host details.

Procedure

1. To the left of the checkbox next to a host name, click to display the storage disks for that host.
2. If there are multiple storage disks for a host, you can select a different disk to act as the installation disk. Click the Role dropdown list for the disk, and then select Installation disk. The role of the previous installation disk changes to None.
3. All bootable disks are marked for reformatting during the installation by default, with the exception of read-only disks such as CDROMs. Deselect the Format checkbox to prevent a disk from being reformatted. The installation disk must be reformatted. Back up any sensitive data before proceeding.

Procedure

1. In the Networking page, select one of the following if it is not already selected for you:

   • Cluster-Managed Networking: Selecting cluster-managed networking means that the Assisted Installer will configure a standard network topology, including keepalived and Virtual Router Redundancy Protocol (VRRP) for managing the API and Ingress VIP addresses.

     Note
     • Currently, Cluster-Managed Networking is not supported on IBM zSystems and IBM Power in OpenShift Container Platform version 4.13.
     • Oracle Cloud Infrastructure (OCI) is available for OpenShift Container Platform 4.14 with a user-managed networking configuration only.
   • User-Managed Networking: Selecting user-managed networking allows you to deploy OpenShift Container Platform with a non-standard network topology. For example, if you want to deploy with an external load balancer instead of keepalived and VRRP, or if you intend to deploy the cluster nodes across many distinct L2 network segments.

2. For cluster-managed networking, configure the following settings:

   1. Define the Machine network. You can use the default network or select a subnet.
   2. Define an API virtual IP. An API virtual IP provides an endpoint for all users to interact with, and configure the platform.
   3. Define an Ingress virtual IP. An Ingress virtual IP provides an endpoint for application traffic flowing from outside the cluster.

3. For user-managed networking, configure the following settings:

   1. Select your Networking stack type:

      • IPv4: Select this type when your hosts are only using IPv4.
      • Dual-stack: You can select dual-stack when your hosts are using IPv4 together with IPv6.
   2. Define the Machine network. You can use the default network or select a subnet.
   3. Define an API virtual IP. An API virtual IP provides an endpoint for all users to interact with, and configure the platform.
   4. Define an Ingress virtual IP. An Ingress virtual IP provides an endpoint for application traffic flowing from outside the cluster.
   5. Optional: You can select Allocate IPs via DHCP server to automatically allocate the API IP and Ingress IP using the DHCP server.

4. Optional: Select Use advanced networking to configure the following advanced networking properties:

   • Cluster network CIDR: Define an IP address block from which Pod IP addresses are allocated.
   • Cluster network host prefix: Define a subnet prefix length to assign to each node.
   • Service network CIDR: Define an IP address to use for service IP addresses.
   • Network type: Select either Software-Defined Networking (SDN) for standard networking or Open Virtual Networking (OVN) for IPv6, dual-stack networking, and telco features. In OpenShift Container Platform 4.12 and later releases, OVN is the default Container Network Interface (CNI).

Procedure

1. Press Begin installation.
2. Click on the link in the Status column of the Host Inventory list to see the installation status of a particular host.

Procedure

1. Make a copy of the kubeadmin username and password.

2. Download the kubeconfig file and copy it to the auth directory under your working directory:

   $ mkdir -p <working_directory>/auth

   $ cp kubeadmin <working_directory>/auth

   Note

   The kubeconfig file is available for download for 24 hours after completing the installation.

3. Add the kubeconfig file to your environment:

   $ export KUBECONFIG=<your working directory>/auth/kubeconfig

4. Login with the oc CLI tool:

   $ oc login -u kubeadmin -p <password>

   Replace <password> with the password of the kubeadmin user.

5. Click on the web console URL or click Launch OpenShift Console to open the console.
6. Enter the kubeadmin username and password. Follow the instructions in the OpenShift Container Platform console to configure an identity provider and configure alert receivers.
7. Add a bookmark of the OpenShift Container Platform console.
8. Complete any post-installation platform integration steps.

Procedure

1. In the menu, click OpenShift.
2. In the submenu, click Downloads.
3. In the Tokens section under OpenShift Cluster Manager API Token, click View API Token.

4. Click Load Token.

   Important

   Disable pop-up blockers.

5. In the Your API token section, copy the offline token.

6. In your terminal, set the offline token to the OFFLINE_TOKEN variable:

   $ export OFFLINE_TOKEN=<copied_api_token>

   Tip

   To make the offline token permanent, add it to your profile.

7. Click Download ocm CLI.
8. Copy the downloaded file to your path. For example, copy the file to /usr/bin or ~/.local/bin and create an ocm symbolic link.

9. Copy and paste the authentication command to your terminal and press Enter to login:

   $ ocm login --token="${OFFLINE_TOKEN}"

Procedure

1. Set the API_TOKEN variable using the OFFLINE_TOKEN to validate the user.

   1. (Optional) On the command line terminal, execute the following command:

      $ export API_TOKEN=$( \
        curl \
        --silent \
        --header "Accept: application/json" \
        --header "Content-Type: application/x-www-form-urlencoded" \
        --data-urlencode "grant_type=refresh_token" \
        --data-urlencode "client_id=cloud-services" \
        --data-urlencode "refresh_token=${OFFLINE_TOKEN}" \
        "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \
        | jq --raw-output ".access_token" \
      )

   2. (Optional) On the command line terminal, login to the ocm client:

      $ ocm login --token="${OFFLINE_TOKEN}"

      Then, generate an API token:

      $ export API_TOKEN=$(ocm token)

2. Create a script in your path for one of the token generating methods. For example:

   $ vim ~/.local/bin/refresh-token

   export API_TOKEN=$( \
     curl \
     --silent \
     --header "Accept: application/json" \
     --header "Content-Type: application/x-www-form-urlencoded" \
     --data-urlencode "grant_type=refresh_token" \
     --data-urlencode "client_id=cloud-services" \
     --data-urlencode "refresh_token=${OFFLINE_TOKEN}" \
     "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \
     | jq --raw-output ".access_token" \
   )

   Then, save the file.

3. Change the file mode to make it executable:

   $ chmod +x ~/.local/bin/refresh-token

4. Refresh the API token:

   $ source refresh-token

5. Verify that you can access the API by running the following command:

   $ curl -s https://api.openshift.com/api/assisted-install/v2/component-versions -H "Authorization: Bearer ${API_TOKEN}" | jq

   Example output

   {
     "release_tag": "v2.11.3",
     "versions": {
       "assisted-installer": "registry.redhat.io/rhai-tech-preview/assisted-installer-rhel8:v1.0.0-211",
       "assisted-installer-controller": "registry.redhat.io/rhai-tech-preview/assisted-installer-reporter-rhel8:v1.0.0-266",
       "assisted-installer-service": "quay.io/app-sre/assisted-service:78d113a",
       "discovery-agent": "registry.redhat.io/rhai-tech-preview/assisted-installer-agent-rhel8:v1.0.0-195"
     }
   }

Procedure

1. In the menu, click OpenShift.
2. In the submenu, click Downloads.
3. In the Tokens section under Pull secret, click Download.

4. To use the pull secret from a shell variable, execute the following command:

   $ export PULL_SECRET=$(cat ~/Downloads/pull-secret.txt | jq -R .)

5. To slurp the pull secret file using jq, reference it in the pull_secret variable, piping the value to tojson to ensure that it is properly formatted as escaped JSON. For example:

   $ curl https://api.openshift.com/api/assisted-install/v2/clusters \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
           --slurpfile pull_secret ~/Downloads/pull-secret.txt ' 

   1

   
       {
           "name": "testcluster",
           "high_availability_mode": "None",
           "openshift_version": "4.11",
           "pull_secret": $pull_secret[0] | tojson, 

   2

   
           "base_dns_domain": "example.com"
       }
   ')"

   1

   Slurp the pull secret file.

   2

   Format the pull secret to escaped JSON format.

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Register a new cluster.

   1. Optional: You can register a new cluster by slurping the pull secret file in the request:

      $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d "$(jq --null-input \
          --slurpfile pull_secret ~/Downloads/pull-secret.txt '
      {
          "name": "testcluster",
          "openshift_version": "4.11",
          "cpu_architecture" : "<architecture_name>" 

      1

      
          "high_availability_mode": <cluster_type>, 

      2

      
          "base_dns_domain": "example.com",
          "pull_secret": $pull_secret[0] | tojson
      }
      ')" | jq '.id'

      Note

      1

      Use any of the following values: x86_64, arm64, ppc64le, s390x, multi. For a mixed-architecture cluster, only use multi.

      2

      Use the default value full to represent a multi-node OpenShift cluster, or none to represent a single-node OpenShift cluster.

   2. Optional: You can register a new cluster by writing the configuration to a JSON file and then referencing it in the request:

      cat << EOF > cluster.json
      {
        "name": "testcluster",
        "openshift_version": "4.11",
        "high_availability_mode": "<cluster_type>",
        "base_dns_domain": "example.com",
        "pull_secret": $PULL_SECRET
      }
      EOF

      $ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters" \
        -d @./cluster.json \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
        | jq '.id'

3. Assign the returned cluster_id to the CLUSTER_ID variable and export it:

   $ export CLUSTER_ID=<cluster_id>

   Note

   If you close your terminal session, you need to export the CLUSTER_ID variable again in a new terminal session.

4. Check the status of the new cluster:

   $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $API_TOKEN" \
     | jq

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Modify the cluster. For example:

   $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
   {
       "ssh_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZrD4LMkAEeoU2vShhF8VM+cCZtVRgB7tqtsMxms2q3TOJZAgfuqReKYWm+OLOZTD+DO3Hn1pah/mU3u7uJfTUg4wEX0Le8zBu9xJVym0BVmSFkzHfIJVTn6SfZ81NqcalisGWkpmkKXVCdnVAX6RsbHfpGKk9YPQarmRCn5KzkelJK4hrSWpBPjdzkFXaIpf64JBZtew9XVYA3QeXkIcFuq7NBuUH9BonroPEmIXNOa41PUP1IWq3mERNgzHZiuU8Ks/pFuU5HCMvv4qbTOIhiig7vidImHPpqYT/TCkuVi5w0ZZgkkBeLnxWxH0ldrfzgFBYAxnpTU8Ih/4VhG538Ix1hxPaM6cXds2ic71mBbtbSrk+zjtNPaeYk1O7UpcCw4jjHspU/rVV/DY51D5gSiiuaFPBMucnYPgUxy4FMBFfGrmGLIzTKiLzcz0DiSz1jBeTQOX++1nz+KDLBD8CPdi5k4dq7lLkapRk85qdEvgaG5RlHMSPSS3wDrQ51fD8= user@hostname"
   }
   ' | jq

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Register a new infrastructure environment. Provide a name, preferably something including the cluster name. This example provides the cluster ID to associate the infrastructure environment with the cluster resource. The following example specifies the image_type. You can specify either full-iso or minimal-iso. The default value is minimal-iso.

   1. Optional: You can register a new infrastructure environment by slurping the pull secret file in the request:

      $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d "$(jq --null-input \
        --slurpfile pull_secret ~/Downloads/pull-secret.txt \
        --arg cluster_id ${CLUSTER_ID} '
          {
            "name": "testcluster-infra-env",
            "image_type":"full-iso",
            "cluster_id": $cluster_id,
            "cpu_architecture" : "<architecture_name>" 

      1

      
            "pull_secret": $pull_secret[0] | tojson
          }
      ')" | jq '.id'

      Note

      1

      Indicates the valid values. They are: x86_64, arm64, ppc64le, s390x, multi

   2. Optional: You can register a new infrastructure environment by writing the configuration to a JSON file and then referencing it in the request:

      $ cat << EOF > infra-envs.json
      {
       "name": "testcluster-infra-env",
       "image_type": "full-iso",
       "cluster_id": "$CLUSTER_ID",
       "pull_secret": $PULL_SECRET
      }
      EOF

      $ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/infra-envs" \
       -d @./infra-envs.json \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer $API_TOKEN" \
       | jq '.id'

3. Assign the returned id to the INFRA_ENV_ID variable and export it:

   $ export INFRA_ENV_ID=<id>

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Modify the infrastructure environment:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID} \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
     --slurpfile pull_secret ~/Downloads/pull-secret.txt '
       {
         "image_type":"minimal-iso",
         "pull_secret": $pull_secret[0] | tojson
       }
   ')" | jq

Procedure

1. Configure the discovery image if needed. See Booting hosts using iPXE in Additional Resources.

2. Refresh the API token:

   $ source refresh-token

3. Get the download URL:

   $ curl -H "Authorization: Bearer ${API_TOKEN}" \
   https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-url

4. Download the discovery image:

   $ wget -O discovery.iso '<url>'

   Replace <url> with the download URL from the previous step.

5. Boot the host(s) with the discovery image. If you are installing on a platform and want to integrate with the platform, see the following additional resource for details:
6. Assign a role to host(s).

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Get the host IDs:

   $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
   --header "Content-Type: application/json" \
     -H "Authorization: Bearer $API_TOKEN" \
   | jq '.host_networks[].host_ids'

3. For installations on IBM Z (s390x) with z/VM, additional kernel arguments are required.

   1. To retrieve the hostIDs for the matching nodes, run the following command:

      curl https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/hosts -H "Authorization: Bearer ${API_TOKEN}" | jq '.[]|[.id,.requested_hostname] | join("|")'

   2. To specify the necessary kernel arguments, run the following command:

      curl https://api.stage.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/$1/installer-args \
      -X PATCH \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d '
      {
      "args": [
      "--append-karg", "rd.neednet=1",
      "--append-karg", "ip=10.14.6.3::10.14.6.1:255.255.255.0:master-0.boea3e06.lnxero1.boe:encbdd0:none",
      "--append-karg", "nameserver=10.14.6.1",
      "--append-karg", "ip=[fd00::3]::[fd00::1]:64::encbdd0:none",
      "--append-karg", "nameserver=[fd00::1]",
      "--append-karg", "zfcp.allow_lun_scan=0",
      "--append-karg", "rd.znet=qeth,0.0.bdd0,0.0.bdd1,0.0.bdd2,layer2=1",
      "--append-karg", "rd.dasd=0.0.5235"
      ]
      }
      ' | jq

      Note

      Each host might have specific kernel arguments.

      Example output

      [
        "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5"
      ]

4. Modify the host:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ 

   1

   
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
       {
         "host_role":"worker"
         "host_name" : "worker-1"
       }
   ' | jq

   1

   Replace <host_id> with the ID of the host.

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Install the cluster:

   $ curl -H "Authorization: Bearer $API_TOKEN" \
   -X POST \
   https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/actions/install | jq

3. Complete any post-installation platform integration steps.

Procedure

1. Optional: Using the UI, in the Cluster details step of the user interface wizard, choose to enable TPM v2 encryption on either the control plane nodes, workers, or both.

2. Optional: Using the API, follow the "Modifying hosts" procedure. Set the disk_encryption.enable_on setting to all, masters, or workers. Set the disk_encryption.mode setting to tpmv2.

   1. Refresh the API token:

      $ source refresh-token

   2. Enable TPM v2 encryption:

      $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
      -X PATCH \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d '
      {
        "disk_encryption": {
          "enable_on": "none",
          "mode": "tpmv2"
        }
      }
      ' | jq

      Valid settings for enable_on are all, master, worker, or none.

Procedure

1. Set up a Tang server or access an existing one. See Network-bound disk encryption for instructions. You can set multiple Tang servers, but the Assisted Installer must be able to connect to all of them during installation.

2. On the Tang server, retrieve the thumbprint for the Tang server using tang-show-keys:

   $ tang-show-keys <port>

   Optional: Replace <port> with the port number. The default port number is 80.

   Example thumbprint

   1gYTN_LpU9ZMB35yn5IbADY5OQ0

3. Optional: Retrieve the thumbprint for the Tang server using jose.

   1. Ensure jose is installed on the Tang server:

      $ sudo dnf install jose

   2. On the Tang server, retrieve the thumbprint using jose:

      $ sudo jose jwk thp -i /var/db/tang/<public_key>.jwk

      Replace <public_key> with the public exchange key for the Tang server.

      Example thumbprint

      1gYTN_LpU9ZMB35yn5IbADY5OQ0

4. Optional: In the Cluster details step of the user interface wizard, choose to enable Tang encryption on either the control plane nodes, workers, or both. You will be required to enter URLs and thumbprints for the Tang servers.

5. Optional: Using the API, follow the "Modifying hosts" procedure.

   1. Refresh the API token:

      $ source refresh-token

   2. Set the disk_encryption.enable_on setting to all, masters, or workers. Set the disk_encryption.mode setting to tang. Set disk_encyrption.tang_servers to provide the URL and thumbprint details about one or more Tang servers:

      $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
      -X PATCH \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d '
      {
        "disk_encryption": {
          "enable_on": "all",
          "mode": "tang",
          "tang_servers": "[{\"url\":\"http://tang.example.com:7500\",\"thumbprint\":\"PLjNyRdGw03zlRoGjQYMahSZGu9\"},{\"url\":\"http://tang2.example.com:7500\",\"thumbprint\":\"XYjNyRdGw03zlRoGjQYMahSZGu3\"}]"
        }
      }
      ' | jq

      Valid settings for enable_on are all, master, worker, or none. Within the tang_servers value, comment out the quotes within the object(s).

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Identify the CLUSTER_ID variable by listing the existing clusters, as follows:

   $ curl -s https://api.openshift.com/api/assisted-install/v2/clusters -H "Authorization: Bearer ${API_TOKEN}" | jq '[ .[] | { "name": .name, "id": .id } ]'

   Sample output

   [
     {
       "name": "lvmtest",
       "id": "475358f9-ed3a-442f-ab9e-48fd68bc8188" 

   1

   
     },
     {
       "name": "mcetest",
       "id": "b5259f97-be09-430e-b5eb-d78420ee509a"
     }
   ]

   Note

   1

   The id value is the <cluster_id>.

3. Assign the returned <cluster_id> to the CLUSTER_ID variable and export it:

   $ export CLUSTER_ID=<cluster_id>

4. Update the cluster with the new Operators:

   $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
   {
       "olm_operators": [{"name": "mce"}, {"name": "cnv"}], 

   1

   
   }
   ' | jq '.id'

   Note

   1

   Indicates the Operators to be installed. Valid values include mce, cnv, lvm, and odf. To remove a previously installed Operator, exclude it from the list of values. To remove all previously installed Operators, type "olm_operators": [].

   Sample output

   {
     <various cluster properties>,
     "monitored_operators": [
       {
         "cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
         "name": "console",
         "operator_type": "builtin",
         "status_updated_at": "0001-01-01T00:00:00.000Z",
         "timeout_seconds": 3600
       },
       {
         "cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
         "name": "cvo",
         "operator_type": "builtin",
         "status_updated_at": "0001-01-01T00:00:00.000Z",
         "timeout_seconds": 3600
       },
       {
         "cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
         "name": "mce",
         "namespace": "multicluster-engine",
         "operator_type": "olm",
         "status_updated_at": "0001-01-01T00:00:00.000Z",
         "subscription_name": "multicluster-engine",
         "timeout_seconds": 3600
       },
       {
         "cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
         "name": "cnv",
         "namespace": "openshift-cnv",
         "operator_type": "olm",
         "status_updated_at": "0001-01-01T00:00:00.000Z",
         "subscription_name": "hco-operatorhub",
         "timeout_seconds": 3600
       },
       {
         "cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
         "name": "lvm",
         "namespace": "openshift-local-storage",
         "operator_type": "olm",
         "status_updated_at": "0001-01-01T00:00:00.000Z",
         "subscription_name": "local-storage-operator",
         "timeout_seconds": 4200
       }
     ],
     <more cluster properties>

   Note

   The output is the description of the new cluster state. The monitored_operators property in the output contains Operators of two types:

   • "operator_type": "builtin": Operators of this type are an integral part of OpenShift Container Platform.
   • "operator_type": "olm": Operators of this type are added either manually by a user or automatically due to dependencies. In the example, the lso Operator was added automatically because the cnv Operator requires it.

Procedure

1. Create an Ignition file and specify the configuration specification version:

   $ vim ~/ignition.conf

   {
     "ignition": { "version": "3.1.0" }
   }

2. Add configuration data to the Ignition file. For example, add a password to the core user.

   1. Generate a password hash:

      $ openssl passwd -6

   2. Add the generated password hash to the core user:

      {
        "ignition": { "version": "3.1.0" },
        "passwd": {
          "users": [
            {
              "name": "core",
              "passwordHash": "$6$spam$M5LGSMGyVD.9XOboxcwrsnwNdF4irpJdAWy.1Ry55syyUiUssIzIAHaOrUHr2zg6ruD8YNBPW9kW0H8EnKXyc1"
            }
          ]
        }
      }

3. Save the Ignition file and export it to the IGNITION_FILE variable:

   $ export IGNITION_FILE=~/ignition.conf

Procedure

1. Create an ignition_config_override JSON object and redirect it to a file:

   $ jq -n \
     --arg IGNITION "$(jq -c . $IGNITION_FILE)" \
     '{ignition_config_override: $IGNITION}' \
     > discovery_ignition.json

2. Refresh the API token:

   $ source refresh-token

3. Patch the infrastructure environment:

   $ curl \
     --header "Authorization: Bearer $API_TOKEN" \
     --header "Content-Type: application/json" \
     -XPATCH \
     -d @discovery_ignition.json \
     https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID | jq

   The ignition_config_override object references the Ignition file.

4. Download the updated discovery image.

Procedure

1. On the administration host, insert a USB drive into a USB port.

2. Copy the ISO image to the USB drive, for example:

   # dd if=<path_to_iso> of=<path_to_usb> status=progress

   where:

   <path_to_iso>

   is the relative path to the downloaded discovery ISO file, for example, discovery.iso.

   <path_to_usb>

   is the location of the connected USB drive, for example, /dev/sdb.

   After the ISO is copied to the USB drive, you can use the USB drive to install the Assisted Installer agent on the cluster host.

Procedure

1. Insert the RHCOS discovery ISO USB drive into the target host.
2. Configure the boot drive order in the server firmware settings to boot from the attached discovery ISO, and then reboot the server.

3. Wait for the host to boot up.

   1. For UI installations, on the administration host, return to the browser. Wait for the host to appear in the list of discovered hosts.

   2. For API installations, refresh the token, check the enabled host count, and gather the host IDs:

      $ source refresh-token

      $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
      --header "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
      | jq '.enabled_host_count'

      $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
      --header "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
      | jq '.host_networks[].host_ids'

      Example output

      [
        "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5"
      ]

Procedure

1. Copy the ISO file to an HTTP server accessible in your network.

2. Boot the host from the hosted ISO file, for example:

   1. Call the redfish API to set the hosted ISO as the VirtualMedia boot media by running the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"Image":"<hosted_iso_file>", "Inserted": true}' \
      -H "Content-Type: application/json" \
      -X POST <host_bmc_address>/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia

      Where:

      <bmc_username>:<bmc_password>

      Is the username and password for the target host BMC.

      <hosted_iso_file>

      Is the URL for the hosted installation ISO, for example: http://webserver.example.com/rhcos-live-minimal.iso. The ISO must be accessible from the target host machine.

      <host_bmc_address>

      Is the BMC IP address of the target host machine.

   2. Set the host to boot from the VirtualMedia device by running the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -X PATCH -H 'Content-Type: application/json' \
      -d '{"Boot": {"BootSourceOverrideTarget": "Cd", "BootSourceOverrideMode": "UEFI", "BootSourceOverrideEnabled": "Once"}}' \
      <host_bmc_address>/redfish/v1/Systems/System.Embedded.1

   3. Reboot the host:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"ResetType": "ForceRestart"}' \
      -H 'Content-type: application/json' \
      -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset

   4. Optional: If the host is powered off, you can boot it using the {"ResetType": "On"} switch. Run the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"ResetType": "On"}' -H 'Content-type: application/json' \
      -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset

Procedure

1. Download the iPXE script directly from the UI, or get the iPXE script from the Assisted Installer:

   $ curl \
     --silent \
     --header "Authorization: Bearer $API_TOKEN" \
     https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/downloads/files?file_name=ipxe-script > ipxe-script

   Example

   #!ipxe
   initrd --name initrd http://api.openshift.com/api/assisted-images/images/<infra_env_id>/pxe-initrd?arch=x86_64&image_token=<token_string>&version=4.10
   kernel http://api.openshift.com/api/assisted-images/boot-artifacts/kernel?arch=x86_64&version=4.10 initrd=initrd coreos.live.rootfs_url=http://api.openshift.com/api/assisted-images/boot-artifacts/rootfs?arch=x86_64&version=4.10 random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8"
   boot

2. Download the required artifacts by extracting URLs from the ipxe-script.

   1. Download the initial RAM disk:

      $ awk '/^initrd /{print $NF}' ipxe-script | curl -o initrd.img

   2. Download the linux kernel:

      $ awk '/^kernel /{print $2}' ipxe-script | curl -o kernel

   3. Download the root filesystem:

      $ grep ^kernel ipxe-script | xargs -n1| grep ^coreos.live.rootfs_url | cut -d = -f 2- | curl -o rootfs.img

3. Change the URLs to the different artifacts in the ipxe-script` to match your local HTTP server. For example:

   #!ipxe
   set webserver http://192.168.0.1
   initrd --name initrd $webserver/initrd.img
   kernel $webserver/kernel initrd=initrd coreos.live.rootfs_url=$webserver/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8"
   boot

4. Optional: When installing with RHEL KVM on IBM zSystems you must boot the host by specifying additional kernel arguments

   random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8

   Note

   If you install with iPXE on RHEL KVM, in some circumstances, the VMs on the VM host are not rebooted on first boot and need to be started manually.

5. Optional: When installing on IBM Power you must download intramfs, kernel, and root as follows:

   1. Copy initrd.img and kernel.img to PXE directory `/var/lib/tftpboot/rhcos `
   2. Copy rootfs.img to HTTPD directory `/var/www/html/install `

   3. Add following entry to `/var/lib/tftpboot/boot/grub2/grub.cfg `:

      if [ ${net_default_mac} == fa:1d:67:35:13:20 ]; then
      default=0
      fallback=1
      timeout=1
      menuentry "CoreOS (BIOS)" {
      echo "Loading kernel"
      linux "/rhcos/kernel.img" ip=dhcp rd.neednet=1 ignition.platform.id=metal ignition.firstboot coreos.live.rootfs_url=http://9.114.98.8:8000/install/rootfs.img
      echo "Loading initrd"
      initrd "/rhcos/initrd.img"
      }
      fi

Procedure

1. Go to the Host Discovery tab and scroll down to the Host Inventory table.

2. Select the Auto-assign drop-down for the required host.

   1. Select Control plane node to assign this host a control plane role.
   2. Select Worker to assign this host a worker role.
3. Check the validation status.

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Get the host IDs:

   $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
   --header "Content-Type: application/json" \
     -H "Authorization: Bearer $API_TOKEN" \
   | jq '.host_networks[].host_ids'

   Example output

   [
     "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5"
   ]

3. Modify the host_role setting:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
       {
         "host_role":"worker"
       }
   ' | jq

   Replace <host_id> with the ID of the host.

Prerequisites

1. You have created an infrastructure environment using the API or have created a cluster using the UI.
2. You have your infrastructure environment ID exported in your shell as $INFRA_ENV_ID.
3. You have credentials to use when accessing the API and have exported a token as $API_TOKEN in your shell.
4. You have YAML files with a static network configuration available as server-a.yaml and server-b.yaml.

Procedure

1. Create a temporary file /tmp/request-body.txt with the API request:

   ---
   jq -n --arg NMSTATE_YAML1 "$(cat server-a.yaml)" --arg NMSTATE_YAML2 "$(cat server-b.yaml)" \
   '{
     "static_network_config": [
       {
         "network_yaml": $NMSTATE_YAML1,
         "mac_interface_map": [{"mac_address": "02:00:00:2c:23:a5", "logical_nic_name": "eth0"}, {"mac_address": "02:00:00:68:73:dc", "logical_nic_name": "eth1"}]
       },
       {
         "network_yaml": $NMSTATE_YAML2,
         "mac_interface_map": [{"mac_address": "02:00:00:9f:85:eb", "logical_nic_name": "eth1"}, {"mac_address": "02:00:00:c8:be:9b", "logical_nic_name": "eth0"}]
        }
     ]
   }' >> /tmp/request-body.txt
   ---

2. Refresh the API token:

   $ source refresh-token

3. Send the request to the Assisted Service API endpoint:

   ---
   curl -H "Content-Type: application/json" \
   -X PATCH -d @/tmp/request-body.txt \
   -H "Authorization: Bearer ${API_TOKEN}" \
   https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID
   ---

Procedure

1. Log in to the cluster using the CLI.

2. Check the architecture setting:

   $ oc adm release info -o json | jq .metadata.metadata

   Ensure that the architecture setting is set to 'multi'.

   {
     "release.openshift.io/architecture": "multi"
   }

Procedure

1. Log in to OpenShift Cluster Manager and click the cluster that you want to expand.
2. Click Add hosts and download the discovery ISO for the new host, adding an SSH public key and configuring cluster-wide proxy settings as needed.
3. Optional: Modify ignition files as needed.
4. Boot the target host using the discovery ISO, and wait for the host to be discovered in the console.
5. Select the host role. It can be either a worker or a control plane host.
6. Start the installation.

7. As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host. When prompted, approve the pending CSRs to complete the installation.

   When the host is successfully installed, it is listed as a host in the cluster web console.

Procedure

1. Authenticate against the Assisted Installer REST API and generate an API token for your session. The generated token is valid for 15 minutes only.

2. Set the $API_URL variable by running the following command:

   $ export API_URL=<api_url> 

   1

   1

   Replace <api_url> with the Assisted Installer API URL, for example, https://api.openshift.com

3. Import the cluster by running the following commands:

   1. Set the $CLUSTER_ID variable. Log in to the cluster and run the following command:

      $ export CLUSTER_ID=$(oc get clusterversion -o jsonpath='{.items[].spec.clusterID}')

   2. Set the $CLUSTER_REQUEST variable that is used to import the cluster:

      $ export CLUSTER_REQUEST=$(jq --null-input --arg openshift_cluster_id "$CLUSTER_ID" '{
        "api_vip_dnsname": "<api_vip>", 

      1

      
        "openshift_cluster_id": $CLUSTER_ID,
        "name": "<openshift_cluster_name>" 

      2

      
      }')

      1

      Replace <api_vip> with the hostname for the cluster’s API server. This can be the DNS domain for the API server or the IP address of the single node which the host can reach. For example, api.compute-1.example.com.

      2

      Replace <openshift_cluster_name> with the plain text name for the cluster. The cluster name should match the cluster name that was set during the Day 1 cluster installation.

   3. Import the cluster and set the $CLUSTER_ID variable. Run the following command:

      $ CLUSTER_ID=$(curl "$API_URL/api/assisted-install/v2/clusters/import" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' \
        -d "$CLUSTER_REQUEST" | tee /dev/stderr | jq -r '.id')

4. Generate the InfraEnv resource for the cluster and set the $INFRA_ENV_ID variable by running the following commands:

   1. Download the pull secret file from Red Hat OpenShift Cluster Manager at console.redhat.com.

   2. Set the $INFRA_ENV_REQUEST variable:

      export INFRA_ENV_REQUEST=$(jq --null-input \
          --slurpfile pull_secret <path_to_pull_secret_file> \

      1

      
          --arg ssh_pub_key "$(cat <path_to_ssh_pub_key>)" \

      2

      
          --arg cluster_id "$CLUSTER_ID" '{
        "name": "<infraenv_name>", 

      3

      
        "pull_secret": $pull_secret[0] | tojson,
        "cluster_id": $cluster_id,
        "ssh_authorized_key": $ssh_pub_key,
        "image_type": "<iso_image_type>" 

      4

      
      }')

      1

      Replace <path_to_pull_secret_file> with the path to the local file containing the downloaded pull secret from Red Hat OpenShift Cluster Manager at console.redhat.com.

      2

      Replace <path_to_ssh_pub_key> with the path to the public SSH key required to access the host. If you do not set this value, you cannot access the host while in discovery mode.

      3

      Replace <infraenv_name> with the plain text name for the InfraEnv resource.

      4

      Replace <iso_image_type> with the ISO image type, either full-iso or minimal-iso.

   3. Post the $INFRA_ENV_REQUEST to the /v2/infra-envs API and set the $INFRA_ENV_ID variable:

      $ INFRA_ENV_ID=$(curl "$API_URL/api/assisted-install/v2/infra-envs" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' -d "$INFRA_ENV_REQUEST" | tee /dev/stderr | jq -r '.id')

5. Get the URL of the discovery ISO for the cluster host by running the following command:

   $ curl -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.download_url'

   Example output

   https://api.openshift.com/api/assisted-images/images/41b91e72-c33e-42ee-b80f-b5c5bbf6431a?arch=x86_64&image_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTYwMjYzNzEsInN1YiI6IjQxYjkxZTcyLWMzM2UtNDJlZS1iODBmLWI1YzViYmY2NDMxYSJ9.1EX_VGaMNejMhrAvVRBS7PDPIQtbOOc8LtG8OukE1a4&type=minimal-iso&version=4.12

6. Download the ISO:

   $ curl -L -s '<iso_url>' --output rhcos-live-minimal.iso 

   1

   1

   Replace <iso_url> with the URL for the ISO from the previous step.

7. Boot the new worker host from the downloaded rhcos-live-minimal.iso.

8. Get the list of hosts in the cluster that are not installed. Keep running the following command until the new host shows up:

   $ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.hosts[] | select(.status != "installed").id'

   Example output

   2294ba03-c264-4f11-ac08-2f1bb2f8c296

9. Set the $HOST_ID variable for the new host, for example:

   $ HOST_ID=<host_id> 

   1

   1

   Replace <host_id> with the host ID from the previous step.

10. Check that the host is ready to install by running the following command:

    Note

    Ensure that you copy the entire command including the complete jq expression.

    $ curl -s $API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID -H "Authorization: Bearer ${API_TOKEN}" | jq '
    def host_name($host):
        if (.suggested_hostname // "") == "" then
            if (.inventory // "") == "" then
                "Unknown hostname, please wait"
            else
                .inventory | fromjson | .hostname
            end
        else
            .suggested_hostname
        end;
    
    def is_notable($validation):
        ["failure", "pending", "error"] | any(. == $validation.status);
    
    def notable_validations($validations_info):
        [
            $validations_info // "{}"
            | fromjson
            | to_entries[].value[]
            | select(is_notable(.))
        ];
    
    {
        "Hosts validations": {
            "Hosts": [
                .hosts[]
                | select(.status != "installed")
                | {
                    "id": .id,
                    "name": host_name(.),
                    "status": .status,
                    "notable_validations": notable_validations(.validations_info)
                }
            ]
        },
        "Cluster validations info": {
            "notable_validations": notable_validations(.validations_info)
        }
    }
    ' -r

    Example output

    {
      "Hosts validations": {
        "Hosts": [
          {
            "id": "97ec378c-3568-460c-bc22-df54534ff08f",
            "name": "localhost.localdomain",
            "status": "insufficient",
            "notable_validations": [
              {
                "id": "ntp-synced",
                "status": "failure",
                "message": "Host couldn't synchronize with any NTP server"
              },
              {
                "id": "api-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              },
              {
                "id": "api-int-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              },
              {
                "id": "apps-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              }
            ]
          }
        ]
      },
      "Cluster validations info": {
        "notable_validations": []
      }
    }

11. When the previous command shows that the host is ready, start the installation using the /v2/infra-envs/{infra_env_id}/hosts/{host_id}/actions/install API by running the following command:

    $ curl -X POST -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/hosts/$HOST_ID/actions/install"  -H "Authorization: Bearer ${API_TOKEN}"

12. As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host.

    Important

    You must approve the CSRs to complete the installation.

    Keep running the following API call to monitor the cluster installation:

    $ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq '{
        "Cluster day-2 hosts":
            [
                .hosts[]
                | select(.status != "installed")
                | {id, requested_hostname, status, status_info, progress, status_updated_at, updated_at, infra_env_id, cluster_id, created_at}
            ]
    }'

    Example output

    {
      "Cluster day-2 hosts": [
        {
          "id": "a1c52dde-3432-4f59-b2ae-0a530c851480",
          "requested_hostname": "control-plane-1",
          "status": "added-to-existing-cluster",
          "status_info": "Host has rebooted and no further updates will be posted. Please check console for progress and to possibly approve pending CSRs",
          "progress": {
            "current_stage": "Done",
            "installation_percentage": 100,
            "stage_started_at": "2022-07-08T10:56:20.476Z",
            "stage_updated_at": "2022-07-08T10:56:20.476Z"
          },
          "status_updated_at": "2022-07-08T10:56:20.476Z",
          "updated_at": "2022-07-08T10:57:15.306369Z",
          "infra_env_id": "b74ec0c3-d5b5-4717-a866-5b6854791bd3",
          "cluster_id": "8f721322-419d-4eed-aa5b-61b50ea586ae",
          "created_at": "2022-07-06T22:54:57.161614Z"
        }
      ]
    }

13. Optional: Run the following command to see all the events for the cluster:

    $ curl -s "$API_URL/api/assisted-install/v2/events?cluster_id=$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -c '.[] | {severity, message, event_time, host_id}'

    Example output

    {"severity":"info","message":"Host compute-0: updated status from insufficient to known (Host is ready to be installed)","event_time":"2022-07-08T11:21:46.346Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from known to installing (Installation is in progress)","event_time":"2022-07-08T11:28:28.647Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from installing to installing-in-progress (Starting installation)","event_time":"2022-07-08T11:28:52.068Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Uploaded logs for host compute-0 cluster 8f721322-419d-4eed-aa5b-61b50ea586ae","event_time":"2022-07-08T11:29:47.802Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from installing-in-progress to added-to-existing-cluster (Host has rebooted and no further updates will be posted. Please check console for progress and to possibly approve pending CSRs)","event_time":"2022-07-08T11:29:48.259Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host: compute-0, reached installation stage Rebooting","event_time":"2022-07-08T11:29:48.261Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}

14. Log in to the cluster and approve the pending CSRs to complete the installation.

1. Create and register a multi-architecture cluster.
2. Create an x86_64 infrastructure environment, download the ISO for x86_64, and add the control plane. The control plane must have the x86_64 architecture.
3. Create an arm64, IBM Power or IBM zSystems infrastructure environment, download the ISO for arm64, IBM Power or IBM zSystems, and add the worker nodes.

Main steps

1. Start the procedure for installing OpenShift Container Platform using the API. For details, see Installing with the Assisted Installer API in the Additional Resources section.

2. When you reach the "Registering a new cluster" step of the installation, register the cluster as a multi-architecture cluster:

   $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
      --slurpfile pull_secret ~/Downloads/pull-secret.txt '
   {
      "name": "testcluster",
      "openshift_version": "<version-number>-multi", 

   1

   
      "cpu_architecture" : "multi" 

   2

   
      "high_availability_mode": "full" 

   3

   
      "base_dns_domain": "example.com",
      "pull_secret": $pull_secret[0] | tojson
   }
   ')" | jq '.id'

   Note

   1

   Use the multi- option for the OpenShift version number; for example, "4.12-multi".

   2

   Set the CPU architecture` to "multi".

   3

   Use the full value to indicate Multi-Node OpenShift.

3. When you reach the "Registering a new infrastructure environment" step of the installation, set cpu_architecture to x86_64:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
    --slurpfile pull_secret ~/Downloads/pull-secret.txt \
    --arg cluster_id ${CLUSTER_ID} '
      {
        "name": "testcluster-infra-env",
        "image_type":"full-iso",
        "cluster_id": $cluster_id,
        "cpu_architecture" : "x86_64"
        "pull_secret": $pull_secret[0] | tojson
      }
   ')" | jq '.id'

4. When you reach the "Adding hosts" step of the installation, set host_role to master:

   Note

   For more information, see Assigning Roles to Hosts in Additional Resources.

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
      {
        "host_role":"master"
      }
   ' | jq

5. Download the discovery image for the x86_64 architecture.
6. Boot the x86_64 architecture hosts using the generated discovery image.
7. Start the installation and wait for the cluster to be fully installed.

8. Repeat the "Registering a new infrastructure environment" step of the installation. This time, set cpu_architecture to one of the following: ppc64le (for IBM Power), s390x (for IBM Z), or arm64. For example:

   $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
      --slurpfile pull_secret ~/Downloads/pull-secret.txt '
   {
      "name": "testcluster",
      "openshift_version": "4.12",
      "cpu_architecture" : "arm64"
      "high_availability_mode": "full"
      "base_dns_domain": "example.com",
      "pull_secret": $pull_secret[0] | tojson
   }
   ')" | jq '.id'

9. Repeat the "Adding hosts" step of the installation. This time, set host_role to worker:

   Note

   For more details, see Assigning Roles to Hosts in Additional Resources.

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
      {
        "host_role":"worker"
      }
   ' | jq

10. Download the discovery image for the arm64, ppc64le or s390x architecture.
11. Boot the architecture hosts using the generated discovery image.
12. Start the installation and wait for the cluster to be fully installed.

Procedure

1. Review and approve CSRs

   1. Review the CertificateSigningRequests (CSRs):

      $ oc get csr | grep Pending

      Example output

      csr-5sd59   8m19s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending
      csr-xzqts   10s     kubernetes.io/kubelet-serving                 system:node:worker-6                                                   <none>              Pending

   2. Approve all pending CSRs:

      $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

      Important

      You must approve the CSRs to complete the installation.

2. Confirm the primary node is in Ready status:

   $ oc get nodes

   Example output

   NAME       STATUS   ROLES    AGE     VERSION
   master-0   Ready    master   4h42m   v1.24.0+3882f8f
   worker-1   Ready    worker   4h29m   v1.24.0+3882f8f
   master-2   Ready    master   4h43m   v1.24.0+3882f8f
   master-3   Ready    master   4h27m   v1.24.0+3882f8f
   worker-4   Ready    worker   4h30m   v1.24.0+3882f8f
   master-5   Ready    master   105s    v1.24.0+3882f8f

   Note

   The etcd-operator requires a Machine Custom Resources (CRs) referencing the new node when the cluster runs with a functional Machine API.

3. Link the Machine CR with BareMetalHost and Node:

   1. Create the BareMetalHost CR with a unique .metadata.name value":

      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: custom-master3
        namespace: openshift-machine-api
        annotations:
      spec:
        automatedCleaningMode: metadata
        bootMACAddress: 00:00:00:00:00:02
        bootMode: UEFI
        customDeploy:
          method: install_coreos
        externallyProvisioned: true
        online: true
        userData:
          name: master-user-data-managed
          namespace: openshift-machine-api

      $ oc create -f <filename>

   2. Apply the BareMetalHost CR:

      $ oc apply -f <filename>

   3. Create the Machine CR using the unique .machine.name value:

      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          machine.openshift.io/instance-state: externally provisioned
          metal3.io/BareMetalHost: openshift-machine-api/custom-master3
        finalizers:
        - machine.machine.openshift.io
        generation: 3
        labels:
          machine.openshift.io/cluster-api-cluster: test-day2-1-6qv96
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
        name: custom-master3
        namespace: openshift-machine-api
      spec:
        metadata: {}
        providerSpec:
          value:
            apiVersion: baremetal.cluster.k8s.io/v1alpha1
            customDeploy:
              method: install_coreos
            hostSelector: {}
            image:
              checksum: ""
              url: ""
            kind: BareMetalMachineProviderSpec
            metadata:
              creationTimestamp: null
            userData:
              name: master-user-data-managed

      $ oc create -f <filename>

   4. Apply the Machine CR:

      $ oc apply -f <filename>

   5. Link BareMetalHost, Machine, and Node using the link-machine-and-node.sh script:

      #!/bin/bash
      
      # Credit goes to https://bugzilla.redhat.com/show_bug.cgi?id=1801238.
      # This script will link Machine object and Node object. This is needed
      # in order to have IP address of the Node present in the status of the Machine.
      
      set -x
      set -e
      
      machine="$1"
      node="$2"
      
      if [ -z "$machine" -o -z "$node" ]; then
          echo "Usage: $0 MACHINE NODE"
          exit 1
      fi
      
      uid=$(echo $node | cut -f1 -d':')
      node_name=$(echo $node | cut -f2 -d':')
      
      oc proxy &
      proxy_pid=$!
      function kill_proxy {
          kill $proxy_pid
      }
      trap kill_proxy EXIT SIGINT
      
      HOST_PROXY_API_PATH="http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts"
      
      function wait_for_json() {
          local name
          local url
          local curl_opts
          local timeout
      
          local start_time
          local curr_time
          local time_diff
      
          name="$1"
          url="$2"
          timeout="$3"
          shift 3
          curl_opts="$@"
          echo -n "Waiting for $name to respond"
          start_time=$(date +%s)
          until curl -g -X GET "$url" "${curl_opts[@]}" 2> /dev/null | jq '.' 2> /dev/null > /dev/null; do
              echo -n "."
              curr_time=$(date +%s)
              time_diff=$(($curr_time - $start_time))
              if [[ $time_diff -gt $timeout ]]; then
                  echo "\nTimed out waiting for $name"
                  return 1
              fi
              sleep 5
          done
          echo " Success!"
          return 0
      }
      wait_for_json oc_proxy "${HOST_PROXY_API_PATH}" 10 -H "Accept: application/json" -H "Content-Type: application/json"
      
      addresses=$(oc get node -n openshift-machine-api ${node_name} -o json | jq -c '.status.addresses')
      
      machine_data=$(oc get machine -n openshift-machine-api -o json ${machine})
      host=$(echo "$machine_data" | jq '.metadata.annotations["metal3.io/BareMetalHost"]' | cut -f2 -d/ | sed 's/"//g')
      
      if [ -z "$host" ]; then
          echo "Machine $machine is not linked to a host yet." 1>&2
          exit 1
      fi
      
      # The address structure on the host doesn't match the node, so extract
      # the values we want into separate variables so we can build the patch
      # we need.
      hostname=$(echo "${addresses}" | jq '.[] | select(. | .type == "Hostname") | .address' | sed 's/"//g')
      ipaddr=$(echo "${addresses}" | jq '.[] | select(. | .type == "InternalIP") | .address' | sed 's/"//g')
      
      host_patch='
      {
        "status": {
          "hardware": {
            "hostname": "'${hostname}'",
            "nics": [
              {
                "ip": "'${ipaddr}'",
                "mac": "00:00:00:00:00:00",
                "model": "unknown",
                "speedGbps": 10,
                "vlanId": 0,
                "pxe": true,
                "name": "eth1"
              }
            ],
            "systemVendor": {
              "manufacturer": "Red Hat",
              "productName": "product name",
              "serialNumber": ""
            },
            "firmware": {
              "bios": {
                "date": "04/01/2014",
                "vendor": "SeaBIOS",
                "version": "1.11.0-2.el7"
              }
            },
            "ramMebibytes": 0,
            "storage": [],
            "cpu": {
              "arch": "x86_64",
              "model": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz",
              "clockMegahertz": 2199.998,
              "count": 4,
              "flags": []
            }
          }
        }
      }
      '
      
      echo "PATCHING HOST"
      echo "${host_patch}" | jq .
      
      curl -s \
           -X PATCH \
           ${HOST_PROXY_API_PATH}/${host}/status \
           -H "Content-type: application/merge-patch+json" \
           -d "${host_patch}"
      
      oc get baremetalhost -n openshift-machine-api -o yaml "${host}"

      $ bash link-machine-and-node.sh custom-master3 worker-5

4. Confirm etcd members:

   $ oc rsh -n openshift-etcd etcd-worker-2
   etcdctl member list -w table

   Example output

   +--------+---------+--------+--------------+--------------+---------+
   |   ID   |  STATUS |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
   +--------+---------+--------+--------------+--------------+---------+
   |2c18942f| started |worker-3|192.168.111.26|192.168.111.26|  false  |
   |61e2a860| started |worker-2|192.168.111.25|192.168.111.25|  false  |
   |ead4f280| started |worker-5|192.168.111.28|192.168.111.28|  false  |
   +--------+---------+--------+--------------+--------------+---------+

5. Confirm the etcd-operator configuration applies to all nodes:

   $ oc get clusteroperator etcd

   Example output

   NAME   VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   etcd   4.11.5    True        False         False      5h54m

6. Confirm etcd-operator health:

   $ oc rsh -n openshift-etcd etcd-worker-0
   etcdctl endpoint health

   Example output

   192.168.111.26 is healthy: committed proposal: took = 11.297561ms
   192.168.111.25 is healthy: committed proposal: took = 13.892416ms
   192.168.111.28 is healthy: committed proposal: took = 11.870755ms

7. Confirm node health:

   $ oc get Nodes

   Example output

   NAME       STATUS   ROLES    AGE     VERSION
   master-0   Ready    master   6h20m   v1.24.0+3882f8f
   worker-1   Ready    worker   6h7m    v1.24.0+3882f8f
   master-2   Ready    master   6h20m   v1.24.0+3882f8f
   master-3   Ready    master   6h4m    v1.24.0+3882f8f
   worker-4   Ready    worker   6h7m    v1.24.0+3882f8f
   master-5   Ready    master   99m     v1.24.0+3882f8f

8. Confirm the ClusterOperators health:

   $ oc get ClusterOperators

   Example output

   NAME                                      VERSION AVAILABLE PROGRESSING DEGRADED SINCE MSG
   authentication                            4.11.5  True      False       False    5h57m
   baremetal                                 4.11.5  True      False       False    6h19m
   cloud-controller-manager                  4.11.5  True      False       False    6h20m
   cloud-credential                          4.11.5  True      False       False    6h23m
   cluster-autoscaler                        4.11.5  True      False       False    6h18m
   config-operator                           4.11.5  True      False       False    6h19m
   console                                   4.11.5  True      False       False    6h4m
   csi-snapshot-controller                   4.11.5  True      False       False    6h19m
   dns                                       4.11.5  True      False       False    6h18m
   etcd                                      4.11.5  True      False       False    6h17m
   image-registry                            4.11.5  True      False       False    6h7m
   ingress                                   4.11.5  True      False       False    6h6m
   insights                                  4.11.5  True      False       False    6h12m
   kube-apiserver                            4.11.5  True      False       False    6h16m
   kube-controller-manager                   4.11.5  True      False       False    6h16m
   kube-scheduler                            4.11.5  True      False       False    6h16m
   kube-storage-version-migrator             4.11.5  True      False       False    6h19m
   machine-api                               4.11.5  True      False       False    6h15m
   machine-approver                          4.11.5  True      False       False    6h19m
   machine-config                            4.11.5  True      False       False    6h18m
   marketplace                               4.11.5  True      False       False    6h18m
   monitoring                                4.11.5  True      False       False    6h4m
   network                                   4.11.5  True      False       False    6h20m
   node-tuning                               4.11.5  True      False       False    6h18m
   openshift-apiserver                       4.11.5  True      False       False    6h8m
   openshift-controller-manager              4.11.5  True      False       False    6h7m
   openshift-samples                         4.11.5  True      False       False    6h12m
   operator-lifecycle-manager                4.11.5  True      False       False    6h18m
   operator-lifecycle-manager-catalog        4.11.5  True      False       False    6h19m
   operator-lifecycle-manager-pkgsvr         4.11.5  True      False       False    6h12m
   service-ca                                4.11.5  True      False       False    6h19m
   storage                                   4.11.5  True      False       False    6h19m

9. Confirm the ClusterVersion:

   $ oc get ClusterVersion

   Example output

   NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
   version   4.11.5    True        False         5h57m   Cluster version is 4.11.5

10. Remove the old control plane node:

    1. Delete the BareMetalHost CR:

       $ oc delete bmh -n openshift-machine-api custom-master3

    2. Confirm the Machine is unhealthy:

       $ oc get machine -A

       Example output

       NAMESPACE              NAME                              PHASE    AGE
       openshift-machine-api  custom-master3                    Running  14h
       openshift-machine-api  test-day2-1-6qv96-master-0        Failed   20h
       openshift-machine-api  test-day2-1-6qv96-master-1        Running  20h
       openshift-machine-api  test-day2-1-6qv96-master-2        Running  20h
       openshift-machine-api  test-day2-1-6qv96-worker-0-8w7vr  Running  19h
       openshift-machine-api  test-day2-1-6qv96-worker-0-rxddj  Running  19h

    3. Delete the Machine CR:

       $ oc delete machine -n openshift-machine-api   test-day2-1-6qv96-master-0
       machine.machine.openshift.io "test-day2-1-6qv96-master-0" deleted

    4. Confirm removal of the Node CR:

       $ oc get nodes

       Example output

       NAME       STATUS   ROLES    AGE   VERSION
       worker-1   Ready    worker   19h   v1.24.0+3882f8f
       master-2   Ready    master   20h   v1.24.0+3882f8f
       master-3   Ready    master   19h   v1.24.0+3882f8f
       worker-4   Ready    worker   19h   v1.24.0+3882f8f
       master-5   Ready    master   15h   v1.24.0+3882f8f

11. Check etcd-operator logs to confirm status of the etcd cluster:

    $ oc logs -n openshift-etcd-operator etcd-operator-8668df65d-lvpjf

    Example output

    E0927 07:53:10.597523       1 base_controller.go:272] ClusterMemberRemovalController reconciliation failed: cannot remove member: 192.168.111.23 because it is reported as healthy but it doesn't have a machine nor a node resource

12. Remove the physical machine to allow etcd-operator to reconcile the cluster members:

    $ oc rsh -n openshift-etcd etcd-worker-2
    etcdctl member list -w table; etcdctl endpoint health

    Example output

    +--------+---------+--------+--------------+--------------+---------+
    |   ID   |  STATUS |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
    +--------+---------+--------+--------------+--------------+---------+
    |2c18942f| started |worker-3|192.168.111.26|192.168.111.26|  false  |
    |61e2a860| started |worker-2|192.168.111.25|192.168.111.25|  false  |
    |ead4f280| started |worker-5|192.168.111.28|192.168.111.28|  false  |
    +--------+---------+--------+--------------+--------------+---------+
    192.168.111.26 is healthy: committed proposal: took = 10.458132ms
    192.168.111.25 is healthy: committed proposal: took = 11.047349ms
    192.168.111.28 is healthy: committed proposal: took = 11.414402ms

Procedure

1. Confirm initial state of the cluster:

   $ oc get nodes

   Example output

   NAME       STATUS     ROLES    AGE   VERSION
   worker-1   Ready      worker   20h   v1.24.0+3882f8f
   master-2   NotReady   master   20h   v1.24.0+3882f8f
   master-3   Ready      master   20h   v1.24.0+3882f8f
   worker-4   Ready      worker   20h   v1.24.0+3882f8f
   master-5   Ready      master   15h   v1.24.0+3882f8f

2. Confirm the etcd-operator detects the cluster as unhealthy:

   $ oc logs -n openshift-etcd-operator etcd-operator-8668df65d-lvpjf

   Example output

   E0927 08:24:23.983733       1 base_controller.go:272] DefragController reconciliation failed: cluster is unhealthy: 2 of 3 members are available, worker-2 is unhealthy

3. Confirm the etcdctl members:

   $ oc rsh -n openshift-etcd etcd-worker-3
   etcdctl member list -w table

   Example output

   +--------+---------+--------+--------------+--------------+---------+
   |   ID   | STATUS  |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
   +--------+---------+--------+--------------+--------------+---------+
   |2c18942f| started |worker-3|192.168.111.26|192.168.111.26|  false  |
   |61e2a860| started |worker-2|192.168.111.25|192.168.111.25|  false  |
   |ead4f280| started |worker-5|192.168.111.28|192.168.111.28|  false  |
   +--------+---------+--------+--------------+--------------+---------+

4. Confirm that etcdctl reports an unhealthy member of the cluster:

   $ etcdctl endpoint health

   Example output

   {"level":"warn","ts":"2022-09-27T08:25:35.953Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000680380/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""}
   192.168.111.28 is healthy: committed proposal: took = 12.465641ms
   192.168.111.26 is healthy: committed proposal: took = 12.297059ms
   192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded
   Error: unhealthy cluster

5. Remove the unhealthy control plane by deleting the Machine Custom Resource:

   $ oc delete machine -n openshift-machine-api test-day2-1-6qv96-master-2

   Note

   The Machine and Node Custom Resources (CRs) will not be deleted if the unhealthy cluster cannot run successfully.

6. Confirm that etcd-operator has not removed the unhealthy machine:

   $ oc logs -n openshift-etcd-operator etcd-operator-8668df65d-lvpjf -f

   Example output

   I0927 08:58:41.249222       1 machinedeletionhooks.go:135] skip removing the deletion hook from machine test-day2-1-6qv96-master-2 since its member is still present with any of: [{InternalIP } {InternalIP 192.168.111.26}]

7. Remove the unhealthy etcdctl member manually:

   $ oc rsh -n openshift-etcd etcd-worker-3\
   etcdctl member list -w table

   Example output

   +--------+---------+--------+--------------+--------------+---------+
   |   ID   |  STATUS |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
   +--------+---------+--------+--------------+--------------+---------+
   |2c18942f| started |worker-3|192.168.111.26|192.168.111.26|  false  |
   |61e2a860| started |worker-2|192.168.111.25|192.168.111.25|  false  |
   |ead4f280| started |worker-5|192.168.111.28|192.168.111.28|  false  |
   +--------+---------+--------+--------------+--------------+---------+

8. Confirm that etcdctl reports an unhealthy member of the cluster:

   $ etcdctl endpoint health

   Example output

   {"level":"warn","ts":"2022-09-27T10:31:07.227Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000d6e00/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""}
   192.168.111.28 is healthy: committed proposal: took = 13.038278ms
   192.168.111.26 is healthy: committed proposal: took = 12.950355ms
   192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded
   Error: unhealthy cluster

9. Remove the unhealthy cluster by deleting the etcdctl member Custom Resource:

   $ etcdctl member remove 61e2a86084aafa62

   Example output

   Member 61e2a86084aafa62 removed from cluster 6881c977b97990d7

10. Confirm members of etcdctl by running the following command:

    $ etcdctl member list -w table

    Example output

    +----------+---------+--------+--------------+--------------+-------+
    |    ID    | STATUS  |  NAME  |  PEER ADDRS  | CLIENT ADDRS |LEARNER|
    +----------+---------+--------+--------------+--------------+-------+
    | 2c18942f | started |worker-3|192.168.111.26|192.168.111.26| false |
    | ead4f280 | started |worker-5|192.168.111.28|192.168.111.28| false |
    +----------+---------+--------+--------------+--------------+-------+

11. Review and approve Certificate Signing Requests

    1. Review the Certificate Signing Requests (CSRs):

       $ oc get csr | grep Pending

       Example output

       csr-5sd59   8m19s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending
       csr-xzqts   10s     kubernetes.io/kubelet-serving                 system:node:worker-6                                                   <none>              Pending

    2. Approve all pending CSRs:

       $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve

       Note

       You must approve the CSRs to complete the installation.

12. Confirm ready status of the control plane node:

    $ oc get nodes

    Example output

    NAME       STATUS   ROLES    AGE     VERSION
    worker-1   Ready    worker   22h     v1.24.0+3882f8f
    master-3   Ready    master   22h     v1.24.0+3882f8f
    worker-4   Ready    worker   22h     v1.24.0+3882f8f
    master-5   Ready    master   17h     v1.24.0+3882f8f
    master-6   Ready    master   2m52s   v1.24.0+3882f8f

13. Validate the Machine, Node and BareMetalHost Custom Resources.

    The etcd-operator requires Machine CRs to be present if the cluster is running with the functional Machine API. Machine CRs are displayed during the Running phase when present.

14. Create Machine Custom Resource linked with BareMetalHost and Node.

    Make sure there is a Machine CR referencing the newly added node.

    Important

    Boot-it-yourself will not create BareMetalHost and Machine CRs, so you must create them. Failure to create the BareMetalHost and Machine CRs will generate errors when running etcd-operator.

15. Add BareMetalHost Custom Resource:

    $ oc create bmh -n openshift-machine-api custom-master3

16. Add Machine Custom Resource:

    $ oc create machine -n openshift-machine-api custom-master3

17. Link BareMetalHost, Machine, and Node by running the link-machine-and-node.sh script:

    #!/bin/bash
    
    # Credit goes to https://bugzilla.redhat.com/show_bug.cgi?id=1801238.
    # This script will link Machine object and Node object. This is needed
    # in order to have IP address of the Node present in the status of the Machine.
    
    set -x
    set -e
    
    machine="$1"
    node="$2"
    
    if [ -z "$machine" -o -z "$node" ]; then
        echo "Usage: $0 MACHINE NODE"
        exit 1
    fi
    
    uid=$(echo $node | cut -f1 -d':')
    node_name=$(echo $node | cut -f2 -d':')
    
    oc proxy &
    proxy_pid=$!
    function kill_proxy {
        kill $proxy_pid
    }
    trap kill_proxy EXIT SIGINT
    
    HOST_PROXY_API_PATH="http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts"
    
    function wait_for_json() {
        local name
        local url
        local curl_opts
        local timeout
    
        local start_time
        local curr_time
        local time_diff
    
        name="$1"
        url="$2"
        timeout="$3"
        shift 3
        curl_opts="$@"
        echo -n "Waiting for $name to respond"
        start_time=$(date +%s)
        until curl -g -X GET "$url" "${curl_opts[@]}" 2> /dev/null | jq '.' 2> /dev/null > /dev/null; do
            echo -n "."
            curr_time=$(date +%s)
            time_diff=$(($curr_time - $start_time))
            if [[ $time_diff -gt $timeout ]]; then
                echo "\nTimed out waiting for $name"
                return 1
            fi
            sleep 5
        done
        echo " Success!"
        return 0
    }
    wait_for_json oc_proxy "${HOST_PROXY_API_PATH}" 10 -H "Accept: application/json" -H "Content-Type: application/json"
    
    addresses=$(oc get node -n openshift-machine-api ${node_name} -o json | jq -c '.status.addresses')
    
    machine_data=$(oc get machine -n openshift-machine-api -o json ${machine})
    host=$(echo "$machine_data" | jq '.metadata.annotations["metal3.io/BareMetalHost"]' | cut -f2 -d/ | sed 's/"//g')
    
    if [ -z "$host" ]; then
        echo "Machine $machine is not linked to a host yet." 1>&2
        exit 1
    fi
    
    # The address structure on the host doesn't match the node, so extract
    # the values we want into separate variables so we can build the patch
    # we need.
    hostname=$(echo "${addresses}" | jq '.[] | select(. | .type == "Hostname") | .address' | sed 's/"//g')
    ipaddr=$(echo "${addresses}" | jq '.[] | select(. | .type == "InternalIP") | .address' | sed 's/"//g')
    
    host_patch='
    {
      "status": {
        "hardware": {
          "hostname": "'${hostname}'",
          "nics": [
            {
              "ip": "'${ipaddr}'",
              "mac": "00:00:00:00:00:00",
              "model": "unknown",
              "speedGbps": 10,
              "vlanId": 0,
              "pxe": true,
              "name": "eth1"
            }
          ],
          "systemVendor": {
            "manufacturer": "Red Hat",
            "productName": "product name",
            "serialNumber": ""
          },
          "firmware": {
            "bios": {
              "date": "04/01/2014",
              "vendor": "SeaBIOS",
              "version": "1.11.0-2.el7"
            }
          },
          "ramMebibytes": 0,
          "storage": [],
          "cpu": {
            "arch": "x86_64",
            "model": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz",
            "clockMegahertz": 2199.998,
            "count": 4,
            "flags": []
          }
        }
      }
    }
    '
    
    echo "PATCHING HOST"
    echo "${host_patch}" | jq .
    
    curl -s \
         -X PATCH \
         ${HOST_PROXY_API_PATH}/${host}/status \
         -H "Content-type: application/merge-patch+json" \
         -d "${host_patch}"
    
    oc get baremetalhost -n openshift-machine-api -o yaml "${host}"

    $ bash link-machine-and-node.sh custom-master3 worker-3

18. Confirm members of etcdctl by running the following command:

    $ oc rsh -n openshift-etcd etcd-worker-3
    etcdctl member list -w table

    Example output

    +---------+-------+--------+--------------+--------------+-------+
    |   ID    | STATUS|  NAME  |   PEER ADDRS | CLIENT ADDRS |LEARNER|
    +---------+-------+--------+--------------+--------------+-------+
    | 2c18942f|started|worker-3|192.168.111.26|192.168.111.26| false |
    | ead4f280|started|worker-5|192.168.111.28|192.168.111.28| false |
    | 79153c5a|started|worker-6|192.168.111.29|192.168.111.29| false |
    +---------+-------+--------+--------------+--------------+-------+

19. Confirm the etcd operator has configured all nodes:

    $ oc get clusteroperator etcd

    Example output

    NAME   VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
    etcd   4.11.5    True        False         False      22h

20. Confirm health of etcdctl:

    $ oc rsh -n openshift-etcd etcd-worker-3
    etcdctl endpoint health

    Example output

    192.168.111.26 is healthy: committed proposal: took = 9.105375ms
    192.168.111.28 is healthy: committed proposal: took = 9.15205ms
    192.168.111.29 is healthy: committed proposal: took = 10.277577ms

21. Confirm the health of the nodes:

    $ oc get Nodes

    Example output

    NAME       STATUS   ROLES    AGE   VERSION
    worker-1   Ready    worker   22h   v1.24.0+3882f8f
    master-3   Ready    master   22h   v1.24.0+3882f8f
    worker-4   Ready    worker   22h   v1.24.0+3882f8f
    master-5   Ready    master   18h   v1.24.0+3882f8f
    master-6   Ready    master   40m   v1.24.0+3882f8f

22. Confirm the health of the ClusterOperators:

    $ oc get ClusterOperators

    Example output

    NAME                               VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
    authentication                     4.11.5    True        False         False      150m
    baremetal                          4.11.5    True        False         False      22h
    cloud-controller-manager           4.11.5    True        False         False      22h
    cloud-credential                   4.11.5    True        False         False      22h
    cluster-autoscaler                 4.11.5    True        False         False      22h
    config-operator                    4.11.5    True        False         False      22h
    console                            4.11.5    True        False         False      145m
    csi-snapshot-controller            4.11.5    True        False         False      22h
    dns                                4.11.5    True        False         False      22h
    etcd                               4.11.5    True        False         False      22h
    image-registry                     4.11.5    True        False         False      22h
    ingress                            4.11.5    True        False         False      22h
    insights                           4.11.5    True        False         False      22h
    kube-apiserver                     4.11.5    True        False         False      22h
    kube-controller-manager            4.11.5    True        False         False      22h
    kube-scheduler                     4.11.5    True        False         False      22h
    kube-storage-version-migrator      4.11.5    True        False         False      148m
    machine-api                        4.11.5    True        False         False      22h
    machine-approver                   4.11.5    True        False         False      22h
    machine-config                     4.11.5    True        False         False      110m
    marketplace                        4.11.5    True        False         False      22h
    monitoring                         4.11.5    True        False         False      22h
    network                            4.11.5    True        False         False      22h
    node-tuning                        4.11.5    True        False         False      22h
    openshift-apiserver                4.11.5    True        False         False      163m
    openshift-controller-manager       4.11.5    True        False         False      22h
    openshift-samples                  4.11.5    True        False         False      22h
    operator-lifecycle-manager         4.11.5    True        False         False      22h
    operator-lifecycle-manager-catalog 4.11.5    True        False         False      22h
    operator-lifecycle-manager-pkgsvr  4.11.5    True        False         False      22h
    service-ca                         4.11.5    True        False         False      22h
    storage                            4.11.5    True        False         False      22h

23. Confirm the ClusterVersion:

    $ oc get ClusterVersion

    Example output

    NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
    version   4.11.5    True        False         22h     Cluster version is 4.11.5