Installing OpenShift Container Platform with the Assisted Installer

Procedure

1. Log in to the Red Hat Hybrid Cloud Console.
2. On the Red Hat OpenShift tile, click OpenShift.
3. On the Red Hat OpenShift Container Platform tile, click Create cluster.
4. Click the Datacenter tab.
5. Under Assisted Installer, click Create cluster.
6. Enter a name for the cluster in the Cluster name field.

7. Enter a base domain for the cluster in the Base domain field. All subdomains for the cluster will use this base domain.

   Note

   The base domain must be a valid DNS name. You must not have a wildcard domain set up for the base domain.

8. From the OpenShift version dropdown list, select the version that you want to install and click Select. By default, the dropdown lists the latest OpenShift version. If you need an older version that is not displayed, click Show all available versions at the bottom of the list, and use the search box to find it.

   Important
   • For a mixed-architecture cluster installation, select OpenShift Container Platform 4.12 or later, and use the -multi option. For instructions on installing a mixed-architecture cluster, see Additional resources.
   • For IBM Power® and IBM Z® platforms, only OpenShift Container Platform 4.13 and later is supported.
   • If you are booting from an iSCSI drive, select OpenShift Container Platform version 4.15 or later.

9. Optional: Select Install single-node OpenShift (SNO) if you want to install OpenShift Container Platform on a single node.

   Note

   Currently, single-node OpenShift is not supported on IBM Z® and IBM Power® platforms.

10. Optional: The Assisted Installer already has the pull secret associated to your account. If you want to use a different pull secret, select Edit pull secret.

11. Optional: If you are installing OpenShift Container Platform on a third-party platform, select the platform from the Integrate with external partner platforms list. Valid values are Nutanix, vSphere or Oracle Cloud Infrastructure. Assisted Installer defaults to having no platform integration.

    Note
    • Assisted Installer supports Oracle Cloud Infrastructure (OCI) integration from OpenShift Container Platform 4.14 and later.
    • For details on each of the external partner integrations, see Additional Resources.

12. Optional: Assisted Installer defaults to using x86_64 CPU architecture. If you are installing OpenShift Container Platform on a different architecture, select the architecture to use. Valid values are arm64, ppc64le, and s390x. Remember that some features are not available with arm64, ppc64le, and s390x CPU architectures.

    Important

    For a mixed-architecture cluster installation, use the default x86_64 architecture. This is the mandatory CPU architecture for the control plane nodes. For instructions on installing a mixed-architecture cluster, see Additional resources.

13. Optional: Select Include custom manifests if you have at least one custom manifest to include in the installation. A custom manifest has additional configurations not currently supported in the Assisted Installer. Selecting the checkbox adds the Custom manifests step to the wizard, where you upload the manifests.

    Important
    • If you are installing OpenShift Container Platform on the Oracle Cloud Infrastructure (OCI) third-party platform, it is mandatory to add the custom manifests provided by Oracle.
    • If you have already added custom manifests, clearing the Include custom manifests checkbox automatically deletes them all. You must confirm the deletion.

14. Optional: The Assisted Installer defaults to DHCP networking. If you are using a static IP configuration, bridges or bonds for the cluster nodes instead of DHCP reservations, select Static IP, bridges, and bonds. Selecting the checkbox adds the Static network configurations step to the wizard. For details, see Configuring static networks.

    Important

    A static IP configuration is not supported in the following scenarios:

    • OpenShift Container Platform installations on Oracle Cloud Infrastructure.
    • OpenShift Container Platform installations on iSCSI boot volumes.

15. Optional: If you want to enable encryption of the installation disks, under Enable encryption of installation disks you can select one of the following:

    • For single-node OpenShift, select Control plane node, worker.

    • For multi-node clusters, select Control plane nodes to encrypt the control plane node installation disks. Select Workers to encrypt worker node installation disks.

      Important

      You cannot change the base domain, the single-node OpenShift checkbox, the CPU architecture, the host’s network configuration, or the disk-encryption after installation begins.

Procedure

1. Select the internet protocol version. Valid options are IPv4 and Dual stack (with IPv4 as primary).

1. If the cluster hosts are on a shared VLAN, enter the VLAN ID.

2. Enter the network-wide IP addresses. If you selected Dual stack networking, you must enter both IPv4 and IPv6 addresses.

   1. Enter the cluster network’s IP address range in CIDR notation.
   2. Enter the default gateway IP address.
   3. Enter the DNS server IP address.

3. Enter the host-specific configuration.

   1. If you are only setting a static IP address that uses a single network interface, use the form view to enter the IP address and the MAC address for each host.
   2. If you use multiple interfaces, bonding, or other advanced networking features, use the YAML view and enter the desired network state for each host that uses NMState syntax. Then, add the MAC address and interface name for each host interface used in your network configuration.

Procedure

1. Select one or more from the following options:

   • Install OpenShift Virtualization

   • Install multicluster engine

     You can deploy the multicluster engine with OpenShift Data Foundation on all OpenShift Container Platform clusters.

     Important

     Deploying the multicluster engine without OpenShift Data Foundation results in the following storage configurations:

     • Multi-node cluster: No storage is configured. You must configure storage after the installation.
     • Single-node OpenShift: LVM Storage is installed.
   • Install Logical Volume Manager Storage
   • Install OpenShift Data Foundation
2. Click Next.

Procedure

1. Click the Add hosts button and select the provisioning type.

   1. Select Minimal image file: Provision with virtual media to download a smaller image that will fetch the data needed to boot. The nodes must have virtual media capability. This is the recommended method for x86_64 and arm64 architectures.

      Important

      This option is mandatory in the following scenarios:

      • If you are installing OpenShift Container Platform on Oracle Cloud Infrastructure.
      • If you are installing OpenShift Container Platform on iSCSI boot volumes.
   2. Select Full image file: Provision with physical media to download the larger full image. This is the recommended method for the ppc64le architecture and for the s390x architecture when installing with RHEL KVM.

   3. Select iPXE: Provision from your network server to boot the hosts using iPXE. This is the recommended method on IBM Z® with z/VM nodes and LPAR (both static and DPM). ISO boot is the recommended method on the RHEL KVM installation.

      Note

      If you are installing OpenShift Container Platform on RHEL KVM, in some circumstances, the VMs on the KVM host are not rebooted on first boot and need to be restarted manually.

2. Optional: Activate the Run workloads on control plane nodes switch to schedule workloads to run on control plane nodes, in addition to the default worker nodes.

   Note

   This option is available for clusters of five or more nodes. For clusters of under five nodes, the system runs workloads on the control plane nodes only, by default. For more details, see Configuring schedulable control plane nodes in Additional Resources.

3. Optional: If the cluster hosts are behind a firewall that requires the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, IP address and port for the HTTP and HTTPS URLs of the proxy server.

   Note

   The proxy username and password must be URL-encoded.

4. Optional: Add an SSH public key so that you can connect to the cluster nodes as the core user. Having a login to the cluster nodes can provide you with debugging information during the installation.

   Important

   Do not skip this procedure in production environments, where disaster recovery and debugging is required.

   1. If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
   2. In the SSH public key field, click Browse to upload the id_rsa.pub file containing the SSH public key. Alternatively, drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
5. Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy, or if the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates. Add additional certificates in X.509 format.
6. Configure the discovery image if needed.
7. Optional: If you are installing on a platform and want to integrate with the platform, select Integrate with your virtualization platform. You must boot all hosts and ensure they appear in the host inventory. All the hosts must be on the same platform.
8. Click Generate Discovery ISO or Generate Script File.
9. Download the discovery ISO or iPXE script.
10. Boot the host(s) with the discovery image or iPXE script.

Procedure

1. From the Options (⋮) menu for a host, select Change hostname. If necessary, enter a new name for the host and click Change. You must ensure that each host has a valid and unique hostname.

   Alternatively, from the Actions list, select Change hostname to rename multiple selected hosts. In the Change Hostname dialog, type the new name and include {{n}} to make each hostname unique. Then click Change.

   Note

   You can see the new names appearing in the Preview pane as you type. The name will be identical for all selected hosts, with the exception of a single-digit increment per host.

2. From the Options (⋮) menu, you can select Delete host to delete a host. Click Delete to confirm the deletion.

   Alternatively, from the Actions list, select Delete to delete multiple selected hosts at the same time. Then click Delete hosts.

   Note

   In a regular deployment, a cluster can have three or more hosts, and three of these must be control plane hosts. If you delete a host that is also a control plane, or if you are left with only two hosts, you will get a message saying that the system is not ready. To restore a host, you will need to reboot it from the discovery ISO.

3. From the Options (⋮) menu for the host, optionally select View host events. The events in the list are presented chronologically.

4. For multi-host clusters, in the Role column next to the host name, you can click on the menu to change the role of the host.

   If you do not select a role, the Assisted Installer will assign the role automatically. The minimum hardware requirements for control plane nodes exceed that of worker nodes. If you assign a role to a host, ensure that you assign the control plane role to hosts that meet the minimum hardware requirements.

5. Click the Status link to view hardware, network and operator validations for the host.
6. Click the arrow to the left of a host name to expand the host details.

Procedure

1. Navigate to the Storage page of the wizard.
2. Expand a host to display the associated storage disks.

3. Select Installation disk from the Role list.

   Note

   Multipath devices are automatically discovered and listed in the host’s inventory. To assign a multipath Fibre Channel disk as the installation disk, choose a disk with Drive type set to Multipath, rather than to FC which indicates a single path.

4. When all storage disks return to Ready status, proceed to the next step.

Procedure

1. Navigate to the Storage page of the wizard.
2. Expand a host to display the associated storage disks.
3. Clear Format for a disk.
4. When all storage disks return to Ready status, proceed to the next step.

Procedure

1. In the Networking step, select one of the following if it is not already selected for you:

   • Cluster-Managed Networking: Selecting cluster-managed networking means that the Assisted Installer will configure a standard network topology, including keepalived and Virtual Router Redundancy Protocol (VRRP) for managing the API and Ingress VIP addresses.

     Note
     • Currently, Cluster-Managed Networking is not supported on IBM Z® and IBM Power® in OpenShift Container Platform version 4.13.
     • Oracle Cloud Infrastructure (OCI) is available for OpenShift Container Platform 4.14 with a user-managed networking configuration only.
   • User-Managed Networking: Selecting user-managed networking deploys OpenShift Container Platform with a non-standard network topology. Select user-managed networking if you want to deploy with an external load balancer instead of keepalived and VRRP, or if you intend to deploy the cluster nodes across many distinct L2 network segments.

2. For cluster-managed networking, configure the following settings:

   1. Define the Machine network. You can use the default network or select a subnet.

      Important

      For iSCSI boot volumes, the hosts are connected over two machine networks: one designated for the OpenShift Container Platform installation and the other for iSCSI traffic. Ensure you select the OpenShift Container Platform network from the dropdown list. Choosing the iSCSI network will result in an Insufficient status for the host in the Networking step.

   2. Define an API virtual IP. An API virtual IP provides an endpoint for all users to interact with, and configure the platform.
   3. Define an Ingress virtual IP. An Ingress virtual IP provides an endpoint for application traffic flowing from outside the cluster.

3. For user-managed networking, configure the following settings:

   1. Select your Networking stack type:

      • IPv4: Select this type when your hosts are only using IPv4.
      • Dual-stack: You can select dual-stack when your hosts are using IPv4 together with IPv6.

   2. Define the Machine network. You can use the default network or select a subnet.

      Important

      For iSCSI boot volumes, the hosts are connected over two machine networks: one designated for the OpenShift Container Platform installation and the other for iSCSI traffic. Ensure you select the OpenShift Container Platform network from the dropdown list. Choosing the iSCSI network will result in an Insufficient status for the host in the Networking step.

   3. Define an API virtual IP. An API virtual IP provides an endpoint for all users to interact with, and configure the platform.
   4. Define an Ingress virtual IP. An Ingress virtual IP provides an endpoint for application traffic flowing from outside the cluster.
   5. Optional: You can select Allocate IPs via DHCP server to automatically allocate the API IP and Ingress IP using the DHCP server.

4. Optional: Select Use advanced networking to configure the following advanced networking properties:

   • Cluster network CIDR: Define an IP address block from which Pod IP addresses are allocated.
   • Cluster network host prefix: Define a subnet prefix length to assign to each node.
   • Service network CIDR: Define an IP address to use for service IP addresses.
   • Network type: Select Open Virtual Networking (OVN) for IPv4, dual-stack networking, and telco features. Starting with OpenShift Container Platform 4.12, OVN is the default Container Network Interface (CNI). Software-Defined Networking (SDN) is supported only up to OpenShift Container Platform 4.14. In OpenShift Container Platform 4.15 and later releases, SDN is no longer supported.

Procedure

1. Click Begin installation.
2. Click the link in the Status column of the Host Inventory list to see the installation status of a particular host.

Procedure

1. Make a copy of the kubeadmin username and password.

2. Download the kubeconfig file and copy it to the auth directory under your working directory:

   $ mkdir -p <working_directory>/auth

   $ cp kubeconfig <working_directory>/auth

   Note

   The kubeconfig file is available for download for 20 days after completing the installation.

3. Add the kubeconfig file to your environment:

   $ export KUBECONFIG=<your working directory>/auth/kubeconfig

4. Login with the oc CLI tool:

   $ oc login -u kubeadmin -p <password>

   Replace <password> with the password of the kubeadmin user.

5. Click the web console URL or click Launch OpenShift Console to open the console.
6. Enter the kubeadmin username and password. Follow the instructions in the OpenShift Container Platform console to configure an identity provider and configure alert receivers.
7. Add a bookmark of the OpenShift Container Platform console.
8. Complete any postinstallation platform integration steps.

Procedure

1. In the menu, click Downloads.
2. In the Tokens section under OpenShift Cluster Manager API Token, click View API Token.

3. Click Load Token.

   Important

   Disable pop-up blockers.

4. In the Your API token section, copy the offline token.

5. In your terminal, set the offline token to the OFFLINE_TOKEN variable:

   $ export OFFLINE_TOKEN=<copied_token>

   Tip

   To make the offline token permanent, add it to your profile.

6. (Optional) Confirm the OFFLINE_TOKEN variable definition.

   $ echo ${OFFLINE_TOKEN}

Procedure

1. On the command line terminal, set the API_TOKEN variable using the OFFLINE_TOKEN to validate the user.

   $ export API_TOKEN=$( \
     curl \
     --silent \
     --header "Accept: application/json" \
     --header "Content-Type: application/x-www-form-urlencoded" \
     --data-urlencode "grant_type=refresh_token" \
     --data-urlencode "client_id=cloud-services" \
     --data-urlencode "refresh_token=${OFFLINE_TOKEN}" \
     "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \
     | jq --raw-output ".access_token" \
   )

2. Confirm the API_TOKEN variable definition:

   $ echo ${API_TOKEN}

3. Create a script in your path for one of the token generating methods. For example:

   $ vim ~/.local/bin/refresh-token

   export API_TOKEN=$( \
     curl \
     --silent \
     --header "Accept: application/json" \
     --header "Content-Type: application/x-www-form-urlencoded" \
     --data-urlencode "grant_type=refresh_token" \
     --data-urlencode "client_id=cloud-services" \
     --data-urlencode "refresh_token=${OFFLINE_TOKEN}" \
     "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \
     | jq --raw-output ".access_token" \
   )

   Then, save the file.

4. Change the file mode to make it executable:

   $ chmod +x ~/.local/bin/refresh-token

5. Refresh the API token:

   $ source refresh-token

6. Verify that you can access the API by running the following command:

   $ curl -s https://api.openshift.com/api/assisted-install/v2/component-versions -H "Authorization: Bearer ${API_TOKEN}" | jq

   Example output

   {
     "release_tag": "v2.11.3",
     "versions": {
       "assisted-installer": "registry.redhat.io/rhai-tech-preview/assisted-installer-rhel8:v1.0.0-211",
       "assisted-installer-controller": "registry.redhat.io/rhai-tech-preview/assisted-installer-reporter-rhel8:v1.0.0-266",
       "assisted-installer-service": "quay.io/app-sre/assisted-service:78d113a",
       "discovery-agent": "registry.redhat.io/rhai-tech-preview/assisted-installer-agent-rhel8:v1.0.0-195"
     }
   }

Procedure

1. In the menu, click OpenShift.
2. In the submenu, click Downloads.
3. In the Tokens section under Pull secret, click Download.

4. To use the pull secret from a shell variable, execute the following command:

   $ export PULL_SECRET=$(cat ~/Downloads/pull-secret.txt | jq -R .)

5. To slurp the pull secret file using jq, reference it in the pull_secret variable, piping the value to tojson to ensure that it is properly formatted as escaped JSON. For example:

   $ curl https://api.openshift.com/api/assisted-install/v2/clusters \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
           --slurpfile pull_secret ~/Downloads/pull-secret.txt ' 1
       {
           "name": "testcluster",
           "high_availability_mode": "None",
           "openshift_version": "4.11",
           "pull_secret": $pull_secret[0] | tojson, 2
           "base_dns_domain": "example.com"
       }
   ')"

   1

   Slurp the pull secret file.

   2

   Format the pull secret to escaped JSON format.

6. Confirm the PULL_SECRET variable definition:

   $ echo ${PULL_SECRET}

Procedure

1. From the root user in your terminal, get the SSH public key:

   $ cat /root/.ssh/id_rsa.pub

2. Set the SSH public key to the CLUSTER_SSHKEY variable:

   $ CLUSTER_SSHKEY=<downloaded_ssh_key>

3. Confirm the CLUSTER_SSHKEY variable definition:

   $ echo ${CLUSTER_SSHKEY}

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Register a new cluster by using one of the following methods:

   • Register the cluster by referencing the pull secret file in the request:

     $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
       -H "Authorization: Bearer ${API_TOKEN}" \
       -H "Content-Type: application/json" \
       -d "$(jq --null-input \
           --slurpfile pull_secret ~/Downloads/pull-secret.txt ' \
       { \
           "name": "testcluster", \
           "openshift_version": "4.16", \ 1
           "high_availability_mode": "<mode>", \ 2
           "cpu_architecture" : "<architecture_name>", \ 3
           "base_dns_domain": "example.com", \
           "pull_secret": $pull_secret[0] | tojson \
       } \
       ')" | jq '.id'

   • Register the cluster by doing the following:

     1. Writing the configuration to a JSON file:

        $ cat << EOF > cluster.json
        {
          "name": "testcluster",
          "openshift_version": "4.16", 1
          "high_availability_mode": "<mode>", 2
          "base_dns_domain": "example.com",
          "network_type": "examplenetwork",
          "cluster_network_cidr":"11.111.1.0/14"
          "cluster_network_host_prefix": 11,
          "service_network_cidr": "111.11.1.0/16",
          "api_vips":[{"ip": ""}],
          "ingress_vips": [{"ip": ""}],
          "vip_dhcp_allocation": false,
          "additional_ntp_source": "clock.redhat.com,clock2.redhat.com",
          "ssh_public_key": "$CLUSTER_SSHKEY",
          "pull_secret": $PULL_SECRET
        }
        EOF

     2. Referencing it in the request:

        $ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters" \
          -d @./cluster.json \
          -H "Content-Type: application/json" \
          -H "Authorization: Bearer $API_TOKEN" \
          | jq '.id'

        1 1

        Pay attention to the following:

        • To install the latest OpenShift version, use the x.y format, such as 4.16 for version 4.16.10. To install a specific OpenShift version, use the x.y.z format, such as 4.16.3 for version 4.16.3.
        • To install a mixed-architecture cluster, add the -multi extension, such as 4.16-multi for the latest version or 4.16.3-multi for a specific version.
        • If you are booting from an iSCSI drive, enter OpenShift Container Platform version 4.15 or later.

        2 2

        Set the value to Full for a high-availability multi-node cluster or None for a single-node OpenShift cluster.

        3

        Valid values are x86_64, arm64, ppc64le, s390x, or multi. Specify multi for a mixed-architecture cluster.

3. Assign the returned cluster_id to the CLUSTER_ID variable and export it:

   $ export CLUSTER_ID=<cluster_id>

   Note

   If you close your terminal session, you need to export the CLUSTER_ID variable again in a new terminal session.

4. Check the status of the new cluster:

   $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $API_TOKEN" \
     | jq

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Modify the cluster. For example, change the SSH key:

   $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
   {
       "ssh_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZrD4LMkAEeoU2vShhF8VM+cCZtVRgB7tqtsMxms2q3TOJZAgfuqReKYWm+OLOZTD+DO3Hn1pah/mU3u7uJfTUg4wEX0Le8zBu9xJVym0BVmSFkzHfIJVTn6SfZ81NqcalisGWkpmkKXVCdnVAX6RsbHfpGKk9YPQarmRCn5KzkelJK4hrSWpBPjdzkFXaIpf64JBZtew9XVYA3QeXkIcFuq7NBuUH9BonroPEmIXNOa41PUP1IWq3mERNgzHZiuU8Ks/pFuU5HCMvv4qbTOIhiig7vidImHPpqYT/TCkuVi5w0ZZgkkBeLnxWxH0ldrfzgFBYAxnpTU8Ih/4VhG538Ix1hxPaM6cXds2ic71mBbtbSrk+zjtNPaeYk1O7UpcCw4jjHspU/rVV/DY51D5gSiiuaFPBMucnYPgUxy4FMBFfGrmGLIzTKiLzcz0DiSz1jBeTQOX++1nz+KDLBD8CPdi5k4dq7lLkapRk85qdEvgaG5RlHMSPSS3wDrQ51fD8= user@hostname"
   }
   ' | jq

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Register a new infrastructure environment. Provide a name, preferably something including the cluster name. This example provides the cluster ID to associate the infrastructure environment with the cluster resource. The following example specifies the image_type. You can specify either full-iso or minimal-iso. The default value is minimal-iso.

   1. Optional: You can register a new infrastructure environment by slurping the pull secret file in the request:

      $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d "$(jq --null-input \
        --slurpfile pull_secret ~/Downloads/pull-secret.txt \
        --arg cluster_id ${CLUSTER_ID} '
          {
            "name": "testcluster-infra-env",
            "image_type":"full-iso",
            "cluster_id": $cluster_id,
            "cpu_architecture" : "<architecture_name>", 1
            "pull_secret": $pull_secret[0] | tojson
          }
      ')" | jq '.id'

      Note

      1

      Valid values are x86_64, arm64, ppc64le, s390x, and multi.

   2. Optional: You can register a new infrastructure environment by writing the configuration to a JSON file and then referencing it in the request:

      $ cat << EOF > infra-envs.json
      {
       "name": "testcluster",
       "pull_secret": $PULL_SECRET,
       "proxy": {
          "http_proxy": "",
          "https_proxy": "",
          "no_proxy": ""
        },
        "ssh_authorized_key": "$CLUSTER_SSHKEY",
        "image_type": "full-iso",
        "cluster_id": "${CLUSTER_ID}",
        "openshift_version": "4.11"
      }
      EOF

      $ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/infra-envs"
       -d @./infra-envs.json
       -H "Content-Type: application/json"
       -H "Authorization: Bearer $API_TOKEN"
       | jq '.id'

3. Assign the returned id to the INFRA_ENV_ID variable and export it:

   $ export INFRA_ENV_ID=<id>

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Modify the infrastructure environment:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID} \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
     --slurpfile pull_secret ~/Downloads/pull-secret.txt '
       {
         "image_type":"minimal-iso",
         "pull_secret": $pull_secret[0] | tojson
       }
   ')" | jq

Procedure

1. Configure the discovery image if needed. For details, see Configuring the discovery image.

2. Refresh the API token:

   $ source refresh-token

3. Get the download URL:

   $ curl -H "Authorization: Bearer ${API_TOKEN}" \
   https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-url

   Example output

   {
     "expires_at": "2024-02-07T20:20:23.000Z",
     "url": "https://api.openshift.com/api/assisted-images/bytoken/<TOKEN>/<OCP_VERSION>/<CPU_ARCHITECTURE>/<FULL_OR_MINIMAL_IMAGE>.iso"
   }

4. Download the discovery image:

   $ wget -O discovery.iso <url>

   Replace <url> with the download URL from the previous step.

5. Boot the host(s) with the discovery image.
6. Assign a role to host(s).

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Get the host IDs:

   $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
   --header "Content-Type: application/json" \
     -H "Authorization: Bearer $API_TOKEN" \
   | jq '.host_networks[].host_ids'

3. Modify the host:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ 1
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
       {
         "host_role":"worker"
         "host_name" : "worker-1"
       }
   ' | jq

   1

   Replace <host_id> with the ID of the host.

Procedure

1. Create a custom manifest file.
2. Save the custom manifest file using the appropriate extension for the file format.

3. Refresh the API token:

   $ source refresh-token

4. Add the custom manifest to the cluster by executing the following command:

   $ curl -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests" \
       -H "Authorization: Bearer $API_TOKEN" \
       -H "Content-Type: application/json" \
       -d '{
               "file_name":"manifest.json",
               "folder":"manifests",
               "content":"'"$(base64 -w 0 ~/manifest.json)"'"
       }' | jq

   Replace manifest.json with the name of your manifest file. The second instance of manifest.json is the path to the file. Ensure the path is correct.

   Example output

   {
     "file_name": "manifest.json",
     "folder": "manifests"
   }

   Note

   The base64 -w 0 command base64-encodes the manifest as a string and omits carriage returns. Encoding with carriage returns will generate an exception.

5. Verify that the Assisted Installer added the manifest:

   $ curl -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests/files?folder=manifests&file_name=manifest.json" -H "Authorization: Bearer $API_TOKEN"

   Replace manifest.json with the name of your manifest file.

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Install the cluster:

   $ curl -H "Authorization: Bearer $API_TOKEN" \
   -X POST \
   https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/actions/install | jq

3. Complete any postinstallation platform integration steps.

Procedure

1. Optional: Using the web console, in the Cluster details step of the user interface wizard, choose to enable TPM v2 encryption on either the control plane nodes, workers, or both.

2. Optional: Using the API, follow the "Modifying hosts" procedure. Set the disk_encryption.enable_on setting to all, masters, or workers. Set the disk_encryption.mode setting to tpmv2.

   1. Refresh the API token:

      $ source refresh-token

   2. Enable TPM v2 encryption:

      $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
      -X PATCH \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d '
      {
        "disk_encryption": {
          "enable_on": "none",
          "mode": "tpmv2"
        }
      }
      ' | jq

      Valid settings for enable_on are all, master, worker, or none.

Procedure

1. Set up a Tang server or access an existing one. See Network-bound disk encryption for instructions. You can set multiple Tang servers, but the Assisted Installer must be able to connect to all of them during installation.

2. On the Tang server, retrieve the thumbprint for the Tang server using tang-show-keys:

   $ tang-show-keys <port>

   Optional: Replace <port> with the port number. The default port number is 80.

   Example thumbprint

   1gYTN_LpU9ZMB35yn5IbADY5OQ0

3. Optional: Retrieve the thumbprint for the Tang server using jose.

   1. Ensure jose is installed on the Tang server:

      $ sudo dnf install jose

   2. On the Tang server, retrieve the thumbprint using jose:

      $ sudo jose jwk thp -i /var/db/tang/<public_key>.jwk

      Replace <public_key> with the public exchange key for the Tang server.

      Example thumbprint

      1gYTN_LpU9ZMB35yn5IbADY5OQ0

4. Optional: In the Cluster details step of the user interface wizard, choose to enable Tang encryption on either the control plane nodes, workers, or both. You will be required to enter URLs and thumbprints for the Tang servers.

5. Optional: Using the API, follow the "Modifying hosts" procedure.

   1. Refresh the API token:

      $ source refresh-token

   2. Set the disk_encryption.enable_on setting to all, masters, or workers. Set the disk_encryption.mode setting to tang. Set disk_encyrption.tang_servers to provide the URL and thumbprint details about one or more Tang servers:

      $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
      -X PATCH \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d '
      {
        "disk_encryption": {
          "enable_on": "all",
          "mode": "tang",
          "tang_servers": "[{\"url\":\"http://tang.example.com:7500\",\"thumbprint\":\"PLjNyRdGw03zlRoGjQYMahSZGu9\"},{\"url\":\"http://tang2.example.com:7500\",\"thumbprint\":\"XYjNyRdGw03zlRoGjQYMahSZGu3\"}]"
        }
      }
      ' | jq

      Valid settings for enable_on are all, master, worker, or none. Within the tang_servers value, comment out the quotes within the object(s).

Procedure

1. Log in to the Red Hat Hybrid Cloud Console and follow the instructions for installing OpenShift Container Platform using the Assisted Installer web console. For details, see Installing with the Assisted Installer web console in Additional Resources.
2. When you reach the Host discovery page, click Add hosts.
3. Optionally change the Provisioning type and additional settings as required. All options are compatible with schedulable control planes.
4. Click Generate Discovery ISO to download the ISO.

5. Set Run workloads on control plane nodes to on.

   Note

   For four nodes or less, this switch is activated automatically and cannot be changed.

6. Click Next.

Procedure

1. Follow the instructions for installing Assisted Installer using the Assisted Installer API. For details, see Installing with the Assisted Installer API in Additional Resources.

2. When you reach the step for registering a new cluster, set the schedulable_masters attribute as follows:

   $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
   {
     "schedulable_masters": true 1
   }
   ' | jq

   1

   Enables the scheduling of workloads on the control plane nodes.

Procedure

1. Create an Ignition file and specify the configuration specification version:

   $ vim ~/ignition.conf

   {
     "ignition": { "version": "3.1.0" }
   }

2. Add configuration data to the Ignition file. For example, add a password to the core user.

   1. Generate a password hash:

      $ openssl passwd -6

   2. Add the generated password hash to the core user:

      {
        "ignition": { "version": "3.1.0" },
        "passwd": {
          "users": [
            {
              "name": "core",
              "passwordHash": "$6$spam$M5LGSMGyVD.9XOboxcwrsnwNdF4irpJdAWy.1Ry55syyUiUssIzIAHaOrUHr2zg6ruD8YNBPW9kW0H8EnKXyc1"
            }
          ]
        }
      }

3. Save the Ignition file and export it to the IGNITION_FILE variable:

   $ export IGNITION_FILE=~/ignition.conf

Procedure

1. Create an ignition_config_override JSON object and redirect it to a file:

   $ jq -n \
     --arg IGNITION "$(jq -c . $IGNITION_FILE)" \
     '{ignition_config_override: $IGNITION}' \
     > discovery_ignition.json

2. Refresh the API token:

   $ source refresh-token

3. Patch the infrastructure environment:

   $ curl \
     --header "Authorization: Bearer $API_TOKEN" \
     --header "Content-Type: application/json" \
     -XPATCH \
     -d @discovery_ignition.json \
     https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID | jq

   The ignition_config_override object references the Ignition file.

4. Download the updated discovery image.

Procedure

1. On the administration host, insert a USB drive into a USB port.

2. Copy the ISO image to the USB drive, for example:

   # dd if=<path_to_iso> of=<path_to_usb> status=progress

   where:

   <path_to_iso>

   is the relative path to the downloaded discovery ISO file, for example, discovery.iso.

   <path_to_usb>

   is the location of the connected USB drive, for example, /dev/sdb.

   After the ISO is copied to the USB drive, you can use the USB drive to install the Assisted Installer agent on the cluster host.

Procedure

1. Insert the RHCOS discovery ISO USB drive into the target host.
2. Configure the boot drive order in the server firmware settings to boot from the attached discovery ISO, and then reboot the server.

3. Wait for the host to boot up.

   1. For web console installations, on the administration host, return to the browser. Wait for the host to appear in the list of discovered hosts.

   2. For API installations, refresh the token, check the enabled host count, and gather the host IDs:

      $ source refresh-token

      $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
      --header "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
      | jq '.enabled_host_count'

      $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
      --header "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
      | jq '.host_networks[].host_ids'

      Example output

      [
        "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5"
      ]

Procedure

1. Copy the ISO file to an HTTP server accessible in your network.

2. Boot the host from the hosted ISO file, for example:

   1. Call the redfish API to set the hosted ISO as the VirtualMedia boot media by running the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"Image":"<hosted_iso_file>", "Inserted": true}' \
      -H "Content-Type: application/json" \
      -X POST <host_bmc_address>/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia

      Where:

      <bmc_username>:<bmc_password>

      Is the username and password for the target host BMC.

      <hosted_iso_file>

      Is the URL for the hosted installation ISO, for example: https://example.com/rhcos-live-minimal.iso. The ISO must be accessible from the target host machine.

      <host_bmc_address>

      Is the BMC IP address of the target host machine.

   2. Set the host to boot from the VirtualMedia device by running the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -X PATCH -H 'Content-Type: application/json' \
      -d '{"Boot": {"BootSourceOverrideTarget": "Cd", "BootSourceOverrideMode": "UEFI", "BootSourceOverrideEnabled": "Once"}}' \
      <host_bmc_address>/redfish/v1/Systems/System.Embedded.1

   3. Reboot the host:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"ResetType": "ForceRestart"}' \
      -H 'Content-type: application/json' \
      -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset

   4. Optional: If the host is powered off, you can boot it using the {"ResetType": "On"} switch. Run the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"ResetType": "On"}' -H 'Content-type: application/json' \
      -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset

Procedure

1. Download the iPXE script directly from the web console, or get the iPXE script from the Assisted Installer by running the following command:

   $ curl \
     --silent \
     --header "Authorization: Bearer $API_TOKEN" \
     https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/downloads/files?file_name=ipxe-script > ipxe-script

   Example

   #!ipxe
   initrd --name initrd http://api.openshift.com/api/assisted-images/images/<infra_env_id>/pxe-initrd?arch=x86_64&image_token=<token_string>&version=4.10
   kernel http://api.openshift.com/api/assisted-images/boot-artifacts/kernel?arch=x86_64&version=4.10 initrd=initrd coreos.live.rootfs_url=http://api.openshift.com/api/assisted-images/boot-artifacts/rootfs?arch=x86_64&version=4.10 random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8"
   boot

2. Download the required artifacts by extracting URLs from the ipxe-script:

   1. Download the initial RAM disk by running the following command:

      $ awk '/^initrd /{print $NF}' ipxe-script \
      | xargs curl -o initrd.img -L

   2. Download the Linux kernel by running the following command:

      $ awk '/^kernel /{print $2}' ipxe-script | xargs curl -o kernel -L

   3. Download the root filesystem by running the following command:

      $ grep ^kernel ipxe_script | xargs -n1 | grep ^coreos.live.rootfs_url | cut -d = -f 2,3,4 | xargs curl -o rootfs.img -L

3. Change the URLs to the different artifacts in the ipxe-script to match your local HTTP server. For example:

   #!ipxe
   set webserver http://192.168.0.1
   initrd --name initrd $webserver/initrd.img
   kernel $webserver/kernel initrd=initrd coreos.live.rootfs_url=$webserver/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8"
   boot

4. Optional: When installing with RHEL KVM on IBM Z® you must boot the host by specifying additional kernel arguments:

   random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8

   Note

   When you install with iPXE on RHEL KVM, the VMs on the VM host might not start on the first boot. You must start them manually.

5. Optional: When installing on IBM Power® you must download the initramfs, kernel, and root images as follows:

   1. Copy the initrd.img and kernel.img images to the /var/lib/tftpboot/rhcos PXE directory.
   2. Copy the rootfs.img to the /var/www/html/install HTTPD directory.

   3. Add the following entry to the /var/lib/tftpboot/boot/grub2/grub.cfg directory:

      if [ ${net_default_mac} == fa:1d:67:35:13:20 ]; then
      default=0
      fallback=1
      timeout=1
      menuentry "CoreOS (BIOS)" {
      echo "Loading kernel"
      linux "/rhcos/kernel.img" ip=dhcp rd.neednet=1 ignition.platform.id=metal ignition.firstboot coreos.live.rootfs_url=http://9.114.98.8:8000/install/rootfs.img
      echo "Loading initrd"
      initrd "/rhcos/initrd.img"
      }
      fi

Procedure

1. Go to the Host Discovery tab and scroll down to the Host Inventory table.
2. Select the Auto-assign drop-down for the required host.
3. Select Control plane node to assign this host a control plane role.
4. Select Worker to assign this host a worker role.
5. Check the validation status.

Procedure

1. Refresh the API token:

   $ source refresh-token

2. Get the host IDs:

   $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
   --header "Content-Type: application/json" \
     -H "Authorization: Bearer $API_TOKEN" \
   | jq '.host_networks[].host_ids'

   Example output

   [
     "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5"
   ]

3. Modify the host_role setting:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
       {
         "host_role":"worker"
       }
   ' | jq

   Replace <host_id> with the ID of the host.

Prerequisites

1. You have created an infrastructure environment using the API or have created a cluster using the web console.
2. You have your infrastructure environment ID exported in your shell as $INFRA_ENV_ID.
3. You have credentials to use when accessing the API and have exported a token as $API_TOKEN in your shell.
4. You have YAML files with a static network configuration available as server-a.yaml and server-b.yaml.

Procedure

1. Create a temporary file /tmp/request-body.txt with the API request:

   jq -n --arg NMSTATE_YAML1 "$(cat server-a.yaml)" --arg NMSTATE_YAML2 "$(cat server-b.yaml)" \
   '{
     "static_network_config": [
       {
         "network_yaml": $NMSTATE_YAML1,
         "mac_interface_map": [{"mac_address": "02:00:00:2c:23:a5", "logical_nic_name": "eth0"}, {"mac_address": "02:00:00:68:73:dc", "logical_nic_name": "eth1"}]
       },
       {
         "network_yaml": $NMSTATE_YAML2,
         "mac_interface_map": [{"mac_address": "02:00:00:9f:85:eb", "logical_nic_name": "eth1"}, {"mac_address": "02:00:00:c8:be:9b", "logical_nic_name": "eth0"}]
        }
     ]
   }' >> /tmp/request-body.txt

2. Refresh the API token:

   $ source refresh-token

3. Send the request to the Assisted Service API endpoint:

   $ curl -H "Content-Type: application/json" \
   -X PATCH -d @/tmp/request-body.txt \
   -H "Authorization: Bearer ${API_TOKEN}" \
   https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID

Procedure

1. Log in to the cluster using the CLI.

2. Check that your cluster uses the architecture payload by running the following command:

   $ oc adm release info -o json | jq .metadata.metadata

1. Create and register a multi-architecture cluster.
2. Create an x86_64 infrastructure environment, download the ISO discovery image for x86_64, and add the control plane. The control plane must have the x86_64 architecture.
3. Create an arm64, IBM Power®, or IBM Z® infrastructure environment, download the ISO discovery images for arm64, IBM Power®, or IBM Z®, and add the worker nodes.

Main steps

1. Start the procedure for installing OpenShift Container Platform using the API. For details, see Installing with the Assisted Installer API in the Additional Resources section.

2. When you reach the "Registering a new cluster" step of the installation, register the cluster as a multi-architecture cluster:

   $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
      --slurpfile pull_secret ~/Downloads/pull-secret.txt '
   {
      "name": "testcluster",
      "openshift_version": "<version-number>-multi", 1
      "cpu_architecture" : "multi" 2
      "high_availability_mode": "full" 3
      "base_dns_domain": "example.com",
      "pull_secret": $pull_secret[0] | tojson
   }
   ')" | jq '.id'

   Note

   1

   Use the multi- option for the OpenShift Container Platform version number; for example, "4.12-multi".

   2

   Set the CPU architecture to "multi".

   3

   Use the full value to indicate Multi-Node OpenShift Container Platform.

3. When you reach the "Registering a new infrastructure environment" step of the installation, set cpu_architecture to x86_64:

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
    --slurpfile pull_secret ~/Downloads/pull-secret.txt \
    --arg cluster_id ${CLUSTER_ID} '
      {
        "name": "testcluster-infra-env",
        "image_type":"full-iso",
        "cluster_id": $cluster_id,
        "cpu_architecture" : "x86_64"
        "pull_secret": $pull_secret[0] | tojson
      }
   ')" | jq '.id'

4. When you reach the "Adding hosts" step of the installation, set host_role to master:

   Note

   For more information, see Assigning Roles to Hosts in Additional Resources.

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
      {
        "host_role":"master"
      }
   ' | jq

5. Download the discovery image for the x86_64 architecture.
6. Boot the x86_64 architecture hosts using the generated discovery image.
7. Start the installation and wait for the cluster to be fully installed.

8. Repeat the "Registering a new infrastructure environment" step of the installation. This time, set cpu_architecture to one of the following: ppc64le (for IBM Power®), s390x (for IBM Z®), or arm64. For example:

   $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
      --slurpfile pull_secret ~/Downloads/pull-secret.txt '
   {
      "name": "testcluster",
      "openshift_version": "4.12",
      "cpu_architecture" : "arm64"
      "high_availability_mode": "full"
      "base_dns_domain": "example.com",
      "pull_secret": $pull_secret[0] | tojson
   }
   ')" | jq '.id'

9. Repeat the "Adding hosts" step of the installation. This time, set host_role to worker:

   Note

   For more details, see Assigning Roles to Hosts in Additional Resources.

   $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
   -X PATCH \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d '
      {
        "host_role":"worker"
      }
   ' | jq

10. Download the discovery image for the arm64, ppc64le or s390x architecture.
11. Boot the architecture hosts using the generated discovery image.
12. Start the installation and wait for the cluster to be fully installed.

Procedure

1. Log in to OpenShift Cluster Manager and click the cluster that you want to expand.
2. Click Add hosts and download the discovery ISO for the new host, adding an SSH public key and configuring cluster-wide proxy settings as needed.
3. Optional: Modify ignition files as needed.
4. Boot the target host using the discovery ISO, and wait for the host to be discovered in the console.
5. Select the host role. It can be either a worker or a control plane host.
6. Start the installation.

7. As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host. When prompted, approve the pending CSRs to complete the installation.

   When the host is successfully installed, it is listed as a host in the cluster web console.

Procedure

1. Authenticate against the Assisted Installer REST API and generate an API token for your session. The generated token is valid for 15 minutes only.

2. Set the $API_URL variable by running the following command:

   $ export API_URL=<api_url> 1

   1

   Replace <api_url> with the Assisted Installer API URL, for example, https://api.openshift.com

3. Import the cluster by running the following commands:

   1. Set the $CLUSTER_ID variable:

      1. Log in to the cluster and run the following command:

         $ export CLUSTER_ID=$(oc get clusterversion -o jsonpath='{.items[].spec.clusterID}')

      2. Display the $CLUSTER_ID variable output:

         $ echo ${CLUSTER_ID}

   2. Set the $CLUSTER_REQUEST variable that is used to import the cluster:

      $ export CLUSTER_REQUEST=$(jq --null-input --arg openshift_cluster_id "$CLUSTER_ID" \
      '{
        "api_vip_dnsname": "<api_vip>", 1
        "openshift_cluster_id": "<cluster_id>", 2
        "name": "<openshift_cluster_name>" 3
      }')

      1

      Replace <api_vip> with the hostname for the cluster’s API server. This can be the DNS domain for the API server or the IP address of the single node which the host can reach. For example, api.compute-1.example.com.

      2

      Replace <cluster_id> with the $CLUSTER_ID output from the previous substep.

      3

      Replace <openshift_cluster_name> with the plain text name for the cluster. The cluster name should match the cluster name that was set during the Day 1 cluster installation.

   3. Import the cluster and set the $CLUSTER_ID variable. Run the following command:

      $ CLUSTER_ID=$(curl "$API_URL/api/assisted-install/v2/clusters/import" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' \
        -d "$CLUSTER_REQUEST" | tee /dev/stderr | jq -r '.id')

4. Generate the InfraEnv resource for the cluster and set the $INFRA_ENV_ID variable by running the following commands:

   1. Download the pull secret file from Red Hat OpenShift Cluster Manager at console.redhat.com.

   2. Set the $INFRA_ENV_REQUEST variable:

      export INFRA_ENV_REQUEST=$(jq --null-input \
          --slurpfile pull_secret <path_to_pull_secret_file> \1
          --arg ssh_pub_key "$(cat <path_to_ssh_pub_key>)" \2
          --arg cluster_id "$CLUSTER_ID" '{
        "name": "<infraenv_name>", 3
        "pull_secret": $pull_secret[0] | tojson,
        "cluster_id": $cluster_id,
        "ssh_authorized_key": $ssh_pub_key,
        "image_type": "<iso_image_type>" 4
      }')

      1

      Replace <path_to_pull_secret_file> with the path to the local file containing the downloaded pull secret from Red Hat OpenShift Cluster Manager at console.redhat.com.

      2

      Replace <path_to_ssh_pub_key> with the path to the public SSH key required to access the host. If you do not set this value, you cannot access the host while in discovery mode.

      3

      Replace <infraenv_name> with the plain text name for the InfraEnv resource.

      4

      Replace <iso_image_type> with the ISO image type, either full-iso or minimal-iso.

   3. Post the $INFRA_ENV_REQUEST to the /v2/infra-envs API and set the $INFRA_ENV_ID variable:

      $ INFRA_ENV_ID=$(curl "$API_URL/api/assisted-install/v2/infra-envs" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' -d "$INFRA_ENV_REQUEST" | tee /dev/stderr | jq -r '.id')

5. Get the URL of the discovery ISO for the cluster host by running the following command:

   $ curl -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.download_url'

   Example output

   https://api.openshift.com/api/assisted-images/images/41b91e72-c33e-42ee-b80f-b5c5bbf6431a?arch=x86_64&image_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTYwMjYzNzEsInN1YiI6IjQxYjkxZTcyLWMzM2UtNDJlZS1iODBmLWI1YzViYmY2NDMxYSJ9.1EX_VGaMNejMhrAvVRBS7PDPIQtbOOc8LtG8OukE1a4&type=minimal-iso&version=4.12

6. Download the ISO:

   $ curl -L -s '<iso_url>' --output rhcos-live-minimal.iso 1

   1

   Replace <iso_url> with the URL for the ISO from the previous step.

7. Boot the new worker host from the downloaded rhcos-live-minimal.iso.

8. Get the list of hosts in the cluster that are not installed. Keep running the following command until the new host shows up:

   $ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.hosts[] | select(.status != "installed").id'

   Example output

   2294ba03-c264-4f11-ac08-2f1bb2f8c296

9. Set the $HOST_ID variable for the new host, for example:

   $ HOST_ID=<host_id> 1

   1

   Replace <host_id> with the host ID from the previous step.

10. Check that the host is ready to install by running the following command:

    Note

    Ensure that you copy the entire command including the complete jq expression.

    $ curl -s $API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID -H "Authorization: Bearer ${API_TOKEN}" | jq '
    def host_name($host):
        if (.suggested_hostname // "") == "" then
            if (.inventory // "") == "" then
                "Unknown hostname, please wait"
            else
                .inventory | fromjson | .hostname
            end
        else
            .suggested_hostname
        end;
    
    def is_notable($validation):
        ["failure", "pending", "error"] | any(. == $validation.status);
    
    def notable_validations($validations_info):
        [
            $validations_info // "{}"
            | fromjson
            | to_entries[].value[]
            | select(is_notable(.))
        ];
    
    {
        "Hosts validations": {
            "Hosts": [
                .hosts[]
                | select(.status != "installed")
                | {
                    "id": .id,
                    "name": host_name(.),
                    "status": .status,
                    "notable_validations": notable_validations(.validations_info)
                }
            ]
        },
        "Cluster validations info": {
            "notable_validations": notable_validations(.validations_info)
        }
    }
    ' -r

    Example output

    {
      "Hosts validations": {
        "Hosts": [
          {
            "id": "97ec378c-3568-460c-bc22-df54534ff08f",
            "name": "localhost.localdomain",
            "status": "insufficient",
            "notable_validations": [
              {
                "id": "ntp-synced",
                "status": "failure",
                "message": "Host couldn't synchronize with any NTP server"
              },
              {
                "id": "api-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              },
              {
                "id": "api-int-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              },
              {
                "id": "apps-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              }
            ]
          }
        ]
      },
      "Cluster validations info": {
        "notable_validations": []
      }
    }

11. When the previous command shows that the host is ready, start the installation using the /v2/infra-envs/{infra_env_id}/hosts/{host_id}/actions/install API by running the following command:

    $ curl -X POST -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/hosts/$HOST_ID/actions/install"  -H "Authorization: Bearer ${API_TOKEN}"

12. As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host.

    Important

    You must approve the CSRs to complete the installation.

    Keep running the following API call to monitor the cluster installation:

    $ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq '{
        "Cluster day-2 hosts":
            [
                .hosts[]
                | select(.status != "installed")
                | {id, requested_hostname, status, status_info, progress, status_updated_at, updated_at, infra_env_id, cluster_id, created_at}
            ]
    }'

    Example output

    {
      "Cluster day-2 hosts": [
        {
          "id": "a1c52dde-3432-4f59-b2ae-0a530c851480",
          "requested_hostname": "control-plane-1",
          "status": "added-to-existing-cluster",
          "status_info": "Host has rebooted and no further updates will be posted. Please check console for progress and to possibly approve pending CSRs",
          "progress": {
            "current_stage": "Done",
            "installation_percentage": 100,
            "stage_started_at": "2022-07-08T10:56:20.476Z",
            "stage_updated_at": "2022-07-08T10:56:20.476Z"
          },
          "status_updated_at": "2022-07-08T10:56:20.476Z",
          "updated_at": "2022-07-08T10:57:15.306369Z",
          "infra_env_id": "b74ec0c3-d5b5-4717-a866-5b6854791bd3",
          "cluster_id": "8f721322-419d-4eed-aa5b-61b50ea586ae",
          "created_at": "2022-07-06T22:54:57.161614Z"
        }
      ]
    }

13. Optional: Run the following command to see all the events for the cluster:

    $ curl -s "$API_URL/api/assisted-install/v2/events?cluster_id=$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -c '.[] | {severity, message, event_time, host_id}'

    Example output

    {"severity":"info","message":"Host compute-0: updated status from insufficient to known (Host is ready to be installed)","event_time":"2022-07-08T11:21:46.346Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from known to installing (Installation is in progress)","event_time":"2022-07-08T11:28:28.647Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from installing to installing-in-progress (Starting installation)","event_time":"2022-07-08T11:28:52.068Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Uploaded logs for host compute-0 cluster 8f721322-419d-4eed-aa5b-61b50ea586ae","event_time":"2022-07-08T11:29:47.802Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from installing-in-progress to added-to-existing-cluster (Host has rebooted and no further updates will be posted. Please check console for progress and to possibly approve pending CSRs)","event_time":"2022-07-08T11:29:48.259Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host: compute-0, reached installation stage Rebooting","event_time":"2022-07-08T11:29:48.261Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}

14. Log in to the cluster and approve the pending CSRs to complete the installation.

Procedure

1. In Cluster details, select Nutanix from the Integrate with external partner platforms dropdown list. The Include custom manifest checkbox is optional.
2. In Host discovery, click the Add hosts button.

3. Optional: Add an SSH public key so that you can connect to the Nutanix VMs as the core user. Having a login to the cluster hosts can provide you with debugging information during the installation.

   1. If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
   2. In the SSH public key field, click Browse to upload the id_rsa.pub file containing the SSH public key. Alternatively, drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.

4. Select the required provisioning type.

   Note

   Minimal image file: Provision with virtual media downloads a smaller image that will fetch the data needed to boot.

5. In Networking, select Cluster-managed networking. Nutanix does not support User-managed networking.

   1. Optional: If the cluster hosts are behind a firewall that requires the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, IP address and port for the HTTP and HTTPS URLs of the proxy server.

      Note

      The proxy username and password must be URL-encoded.

   2. Optional: Configure the discovery image if you want to boot it with an ignition file. See Configuring the discovery image for additional details.
6. Click Generate Discovery ISO.
7. Copy the Discovery ISO URL.
8. In the Nutanix Prism UI, follow the directions to upload the discovery image from the Assisted Installer.

9. In the Nutanix Prism UI, create the control plane (master) VMs through Prism Central.

   1. Enter the Name. For example, control-plane or master.
   2. Enter the Number of VMs. This should be 3 for the control plane.
   3. Ensure the remaining settings meet the minimum requirements for control plane hosts.

10. In the Nutanix Prism UI, create the worker VMs through Prism Central.

    1. Enter the Name. For example, worker.
    2. Enter the Number of VMs. You should create at least 2 worker nodes.
    3. Ensure the remaining settings meet the minimum requirements for worker hosts.
11. Return to the Assisted Installer user interface and wait until the Assisted Installer discovers the hosts and each of them have a Ready status.
12. Continue with the installation procedure.

Procedure

1. Configure the discovery image if you want it to boot with an ignition file.

2. Create a Nutanix cluster configuration file to hold the environment variables:

   $ touch ~/nutanix-cluster-env.sh

   $ chmod +x ~/nutanix-cluster-env.sh

   If you have to start a new terminal session, you can reload the environment variables easily. For example:

   $ source ~/nutanix-cluster-env.sh

3. Assign the Nutanix cluster’s name to the NTX_CLUSTER_NAME environment variable in the configuration file:

   $ cat << EOF >> ~/nutanix-cluster-env.sh
   export NTX_CLUSTER_NAME=<cluster_name>
   EOF

   Replace <cluster_name> with the name of the Nutanix cluster.

4. Assign the Nutanix cluster’s subnet name to the NTX_SUBNET_NAME environment variable in the configuration file:

   $ cat << EOF >> ~/nutanix-cluster-env.sh
   export NTX_SUBNET_NAME=<subnet_name>
   EOF

   Replace <subnet_name> with the name of the Nutanix cluster’s subnet.

5. Refresh the API token:

   $ source refresh-token

6. Get the download URL:

   $ curl -H "Authorization: Bearer ${API_TOKEN}" \
   https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-url

7. Create the Nutanix image configuration file:

   $ cat << EOF > create-image.json
   {
     "spec": {
       "name": "ocp_ai_discovery_image.iso",
       "description": "ocp_ai_discovery_image.iso",
       "resources": {
         "architecture": "X86_64",
         "image_type": "ISO_IMAGE",
         "source_uri": "<image_url>",
         "source_options": {
           "allow_insecure_connection": true
         }
       }
     },
     "metadata": {
       "spec_version": 3,
       "kind": "image"
     }
   }
   EOF

   Replace <image_url> with the image URL downloaded from the previous step.

8. Create the Nutanix image:

   $ curl  -k -u <user>:'<password>' -X 'POST' \
   'https://<domain-or-ip>:<port>/api/nutanix/v3/images \
     -H 'accept: application/json' \
     -H 'Content-Type: application/json' \
     -d @./create-image.json | jq '.metadata.uuid'

   Replace <user> with the Nutanix user name. Replace '<password>' with the Nutanix password. Replace <domain-or-ip> with the domain name or IP address of the Nutanix plaform. Replace <port> with the port for the Nutanix server. The port defaults to 9440.

9. Assign the returned UUID to the NTX_IMAGE_UUID environment variable in the configuration file:

   $ cat << EOF >> ~/nutanix-cluster-env.sh
   export NTX_IMAGE_UUID=<uuid>
   EOF

10. Get the Nutanix cluster UUID:

    $ curl -k -u <user>:'<password>' -X 'POST' \
      'https://<domain-or-ip>:<port>/api/nutanix/v3/clusters/list' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "kind": "cluster"
    }'  | jq '.entities[] | select(.spec.name=="<nutanix_cluster_name>") | .metadata.uuid'

    Replace <user> with the Nutanix user name. Replace '<password>' with the Nutanix password. Replace <domain-or-ip> with the domain name or IP address of the Nutanix plaform. Replace <port> with the port for the Nutanix server. The port defaults to 9440. Replace <nutanix_cluster_name> with the name of the Nutanix cluster.

11. Assign the returned Nutanix cluster UUID to the NTX_CLUSTER_UUID environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_CLUSTER_UUID=<uuid>
    EOF

    Replace <uuid> with the returned UUID of the Nutanix cluster.

12. Get the Nutanix cluster’s subnet UUID:

    $ curl -k -u <user>:'<password>' -X 'POST' \
      'https://<domain-or-ip>:<port>/api/nutanix/v3/subnets/list' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "kind": "subnet",
      "filter": "name==<subnet_name>"
    }' | jq '.entities[].metadata.uuid'

    Replace <user> with the Nutanix user name. Replace '<password>' with the Nutanix password. Replace <domain-or-ip> with the domain name or IP address of the Nutanix plaform. Replace <port> with the port for the Nutanix server. The port defaults to 9440. Replace <subnet_name> with the name of the cluster’s subnet.

13. Assign the returned Nutanix subnet UUID to the NTX_CLUSTER_UUID environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_SUBNET_UUID=<uuid>
    EOF

    Replace <uuid> with the returned UUID of the cluster subnet.

14. Ensure the Nutanix environment variables are set:

    $ source ~/nutanix-cluster-env.sh

15. Create a VM configuration file for each Nutanix host. Create three control plane (master) VMs and at least two worker VMs. For example:

    $ touch create-master-0.json

    $ cat << EOF > create-master-0.json
    {
       "spec": {
          "name": "<host_name>",
          "resources": {
             "power_state": "ON",
             "num_vcpus_per_socket": 1,
             "num_sockets": 16,
             "memory_size_mib": 32768,
             "disk_list": [
                {
                   "disk_size_mib": 122880,
                   "device_properties": {
                      "device_type": "DISK"
                   }
                },
                {
                   "device_properties": {
                      "device_type": "CDROM"
                   },
                   "data_source_reference": {
                     "kind": "image",
                     "uuid": "$NTX_IMAGE_UUID"
                  }
                }
             ],
             "nic_list": [
                {
                   "nic_type": "NORMAL_NIC",
                   "is_connected": true,
                   "ip_endpoint_list": [
                      {
                         "ip_type": "DHCP"
                      }
                   ],
                   "subnet_reference": {
                      "kind": "subnet",
                      "name": "$NTX_SUBNET_NAME",
                      "uuid": "$NTX_SUBNET_UUID"
                   }
                }
             ],
             "guest_tools": {
                "nutanix_guest_tools": {
                   "state": "ENABLED",
                   "iso_mount_state": "MOUNTED"
                }
             }
          },
          "cluster_reference": {
             "kind": "cluster",
             "name": "$NTX_CLUSTER_NAME",
             "uuid": "$NTX_CLUSTER_UUID"
          }
       },
       "api_version": "3.1.0",
       "metadata": {
          "kind": "vm"
       }
    }
    EOF

    Replace <host_name> with the name of the host.

16. Boot each Nutanix virtual machine:

    $ curl -k -u <user>:'<password>' -X 'POST' \
      'https://<domain-or-ip>:<port>/api/nutanix/v3/vms' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d @./<vm_config_file_name> | jq '.metadata.uuid'

    Replace <user> with the Nutanix user name. Replace '<password>' with the Nutanix password. Replace <domain-or-ip> with the domain name or IP address of the Nutanix plaform. Replace <port> with the port for the Nutanix server. The port defaults to 9440. Replace <vm_config_file_name> with the name of the VM configuration file.

17. Assign the returned VM UUID to a unique environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_MASTER_0_UUID=<uuid>
    EOF

    Replace <uuid> with the returned UUID of the VM.

    Note

    The environment variable must have a unique name for each VM.

18. Wait until the Assisted Installer has discovered each VM and they have passed validation.

    $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID"
    --header "Content-Type: application/json"
    -H "Authorization: Bearer $API_TOKEN"
    | jq '.enabled_host_count'

19. Modify the cluster definition to enable integration with Nutanix:

    $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
    {
        "platform_type":"nutanix"
    }
    ' | jq

20. Continue with the installation procedure.

Procedure

1. Configure the discovery image if you want it to boot with an ignition file.
2. In Cluster details, select vSphere from the Integrate with external partner platforms dropdown list. The Include custom manifest checkbox is optional.
3. In Host discovery, click the Add hosts button and select the provisioning type.

4. Add an SSH public key so that you can connect to the vSphere VMs as the core user. Having a login to the cluster hosts can provide you with debugging information during the installation.

   1. If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
   2. In the SSH public key field, click Browse to upload the id_rsa.pub file containing the SSH public key. Alternatively, drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.

5. Select the required discovery image ISO.

   Note

   Minimal image file: Provision with virtual media downloads a smaller image that will fetch the data needed to boot.

6. In Networking, select Cluster-managed networking or User-managed networking:

   1. Optional: If the cluster hosts are behind a firewall that requires the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, IP address and port for the HTTP and HTTPS URLs of the proxy server.

      Note

      The proxy username and password must be URL-encoded.

   2. Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy or the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates and add the additional certificates.
   3. Optional: Configure the discovery image if you want to boot it with an ignition file. For more information, see Additional Resources.
7. Click Generate Discovery ISO.
8. Copy the Discovery ISO URL.

9. Download the discovery ISO:

   $ wget - O vsphere-discovery-image.iso <discovery_url>

   Replace <discovery_url> with the Discovery ISO URL from the preceding step.

10. On the command line, power off and delete any preexisting virtual machines:

    $ for VM in $(/usr/local/bin/govc ls /<datacenter>/vm/<folder_name>)
    do
     	/usr/local/bin/govc vm.power -off $VM
     	/usr/local/bin/govc vm.destroy $VM
    done

    Replace <datacenter> with the name of the data center. Replace <folder_name> with the name of the VM inventory folder.

11. Remove preexisting ISO images from the data store, if there are any:

    $ govc datastore.rm -ds <iso_datastore> <image>

    Replace <iso_datastore> with the name of the data store. Replace image with the name of the ISO image.

12. Upload the Assisted Installer discovery ISO:

    $ govc datastore.upload -ds <iso_datastore>  vsphere-discovery-image.iso

    Replace <iso_datastore> with the name of the data store.

    Note

    All nodes in the cluster must boot from the discovery image.

13. Boot three control plane nodes:

    $ govc vm.create -net.adapter <network_adapter_type> \
                     -disk.controller <disk_controller_type> \
                     -pool=<resource_pool> \
                     -c=16 \
                     -m=32768 \
                     -disk=120GB \
                     -disk-datastore=<datastore_file> \
                     -net.address="<nic_mac_address>" \
                     -iso-datastore=<iso_datastore> \
                     -iso="vsphere-discovery-image.iso" \
                     -folder="<inventory_folder>" \
                     <hostname>.<cluster_name>.example.com

    See vm.create for details.

    Note

    The foregoing example illustrates the minimum required resources for control plane nodes.

14. Boot at least two worker nodes:

    $ govc vm.create -net.adapter <network_adapter_type> \
                     -disk.controller <disk_controller_type> \
                     -pool=<resource_pool> \
                     -c=4 \
                     -m=8192 \
                     -disk=120GB \
                     -disk-datastore=<datastore_file> \
                     -net.address="<nic_mac_address>" \
                     -iso-datastore=<iso_datastore> \
                     -iso="vsphere-discovery-image.iso" \
                     -folder="<inventory_folder>" \
                     <hostname>.<cluster_name>.example.com

    See vm.create for details.

    Note

    The foregoing example illustrates the minimum required resources for worker nodes.

15. Ensure the VMs are running:

    $  govc ls /<datacenter>/vm/<folder_name>

    Replace <datacenter> with the name of the data center. Replace <folder_name> with the name of the VM inventory folder.

16. After 2 minutes, shut down the VMs:

    $ for VM in $(govc ls /<datacenter>/vm/<folder_name>)
    do
         govc vm.power -s=true $VM
    done

    Replace <datacenter> with the name of the data center. Replace <folder_name> with the name of the VM inventory folder.

17. Set the disk.enableUUID setting to TRUE:

    $ for VM in $(govc ls /<datacenter>/vm/<folder_name>)
    do
         govc vm.change -vm $VM -e disk.enableUUID=TRUE
    done

    Replace <datacenter> with the name of the data center. Replace <folder_name> with the name of the VM inventory folder.

    Note

    You must set disk.enableUUID to TRUE on all of the nodes to enable autoscaling with vSphere.

18. Restart the VMs:

    $  for VM in $(govc ls /<datacenter>/vm/<folder_name>)
    do
         govc vm.power -on=true $VM
    done

    Replace <datacenter> with the name of the data center. Replace <folder_name> with the name of the VM inventory folder.

19. Return to the Assisted Installer user interface and wait until the Assisted Installer discovers the hosts and each of them have a Ready status.
20. Select roles if needed.
21. In Networking, clear the Allocate IPs via DHCP server checkbox.
22. Set the API VIP address.
23. Set the Ingress VIP address.
24. Continue with the installation procedure.

Procedure

1. Generate a base64-encoded username and password for vCenter:

   $ echo -n "<vcenter_username>" | base64 -w0

   Replace <vcenter_username> with your vCenter username.

   $ echo -n "<vcenter_password>" | base64 -w0

   Replace <vcenter_password> with your vCenter password.

2. Backup the vSphere credentials:

   $ oc get secret vsphere-creds -o yaml -n kube-system > creds_backup.yaml

3. Edit the vSphere credentials:

   $ cp creds_backup.yaml vsphere-creds.yaml

   $ vi vsphere-creds.yaml

   apiVersion: v1
   data:
     <vcenter_address>.username: <vcenter_username_encoded>
     <vcenter_address>.password: <vcenter_password_encoded>
   kind: Secret
   metadata:
     annotations:
       cloudcredential.openshift.io/mode: passthrough
     creationTimestamp: "2022-01-25T17:39:50Z"
     name: vsphere-creds
     namespace: kube-system
     resourceVersion: "2437"
     uid: 06971978-e3a5-4741-87f9-2ca3602f2658
   type: Opaque

   Replace <vcenter_address> with the vCenter address. Replace <vcenter_username_encoded> with the base64-encoded version of your vSphere username. Replace <vcenter_password_encoded> with the base64-encoded version of your vSphere password.

4. Replace the vSphere credentials:

   $ oc replace -f vsphere-creds.yaml

5. Redeploy the kube-controller-manager pods:

   $ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge

6. Backup the vSphere cloud provider configuration:

   $ oc get cm cloud-provider-config -o yaml -n openshift-config > cloud-provider-config_backup.yaml

7. Edit the cloud provider configuration:

   $ cloud-provider-config_backup.yaml cloud-provider-config.yaml

   $ vi cloud-provider-config.yaml

   apiVersion: v1
   data:
     config: |
       [Global]
       secret-name = "vsphere-creds"
       secret-namespace = "kube-system"
       insecure-flag = "1"
   
       [Workspace]
       server = "<vcenter_address>"
       datacenter = "<datacenter>"
       default-datastore = "<datastore>"
       folder = "/<datacenter>/vm/<folder>"
   
       [VirtualCenter "<vcenter_address>"]
       datacenters = "<datacenter>"
   kind: ConfigMap
   metadata:
     creationTimestamp: "2022-01-25T17:40:49Z"
     name: cloud-provider-config
     namespace: openshift-config
     resourceVersion: "2070"
     uid: 80bb8618-bf25-442b-b023-b31311918507

   Replace <vcenter_address> with the vCenter address. Replace <datacenter> with the name of the data center. Replace <datastore> with the name of the data store. Replace <folder> with the folder containing the cluster VMs.

8. Apply the cloud provider configuration:

   $ oc apply -f cloud-provider-config.yaml

9. Taint the nodes with the uninitialized taint:

   Important

   Follow steps 9 through 12 if you are installing OpenShift Container Platform 4.13 or later.

   1. Identify the nodes to taint:

      $ oc get nodes

   2. Run the following command for each node:

      $ oc adm taint node <node_name> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

      Replace <node_name> with the name of the node.

   Example

   $ oc get nodes
   NAME                STATUS   ROLES                  AGE   VERSION
   master-0   Ready    control-plane,master   45h   v1.26.3+379cd9f
   master-1   Ready    control-plane,master   45h   v1.26.3+379cd9f
   worker-0   Ready    worker                 45h   v1.26.3+379cd9f
   worker-1   Ready    worker                 45h   v1.26.3+379cd9f
   master-2   Ready    control-plane,master   45h   v1.26.3+379cd9f
   
   $ oc adm taint node master-0 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
   $ oc adm taint node master-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
   $ oc adm taint node master-2 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
   $ oc adm taint node worker-0 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
   $ oc adm taint node worker-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

10. Back up the infrastructures configuration:

    $ oc get infrastructures.config.openshift.io -o yaml > infrastructures.config.openshift.io.yaml.backup

11. Edit the infrastructures configuration:

    $ cp infrastructures.config.openshift.io.yaml.backup infrastructures.config.openshift.io.yaml

    $ vi infrastructures.config.openshift.io.yaml

    apiVersion: v1
    items:
    - apiVersion: config.openshift.io/v1
      kind: Infrastructure
      metadata:
        creationTimestamp: "2022-05-07T10:19:55Z"
        generation: 1
        name: cluster
        resourceVersion: "536"
        uid: e8a5742c-6d15-44e6-8a9e-064b26ab347d
      spec:
        cloudConfig:
          key: config
          name: cloud-provider-config
        platformSpec:
          type: VSphere
          vsphere:
            failureDomains:
            - name: assisted-generated-failure-domain
              region: assisted-generated-region
              server: <vcenter_address>
              topology:
                computeCluster: /<data_center>/host/<vcenter_cluster>
                datacenter: <data_center>
                datastore: /<data_center>/datastore/<datastore>
                folder: "/<data_center>/path/to/folder"
                networks:
                - "VM Network"
                resourcePool: /<data_center>/host/<vcenter_cluster>/Resources
              zone: assisted-generated-zone
            nodeNetworking:
              external: {}
              internal: {}
            vcenters:
            - datacenters:
              - <data_center>
              server: <vcenter_address>
    
    kind: List
    metadata:
      resourceVersion: ""

    Replace <vcenter_address> with your vCenter address. Replace <datacenter> with the name of your vCenter data center. Replace <datastore> with the name of your vCenter data store. Replace <folder> with the folder containing the cluster VMs. Replace <vcenter_cluster> with the vSphere vCenter cluster where OpenShift Container Platform is installed.

12. Apply the infrastructures configuration:

    $ oc apply -f infrastructures.config.openshift.io.yaml --overwrite=true

Procedure

1. In the Administrator perspective, navigate to Home → Overview.
2. Under Status, click vSphere connection to open the vSphere connection configuration wizard.
3. In the vCenter field, enter the network address of the vSphere vCenter server. This can be either a domain name or an IP address. It appears in the vSphere web client URL; for example https://[your_vCenter_address]/ui.

4. In the vCenter cluster field, enter the name of the vSphere vCenter cluster where OpenShift Container Platform is installed.

   Important

   This step is mandatory if you installed OpenShift Container Platform 4.13 or later.

5. In the Username field, enter your vSphere vCenter username.

6. In the Password field, enter your vSphere vCenter password.

   Warning

   The system stores the username and password in the vsphere-creds secret in the kube-system namespace of the cluster. An incorrect vCenter username or password makes the cluster nodes unschedulable.

7. In the Datacenter field, enter the name of the vSphere data center that contains the virtual machines used to host the cluster; for example, SDDC-Datacenter.

8. In the Default data store field, enter the vSphere data store that stores the persistent data volumes; for example, /SDDC-Datacenter/datastore/datastorename.

   Warning

   Updating the vSphere data center or default data store after the configuration has been saved detaches any active vSphere PersistentVolumes.

9. In the Virtual Machine Folder field, enter the data center folder that contains the virtual machine of the cluster; for example, /SDDC-Datacenter/vm/ci-ln-hjg4vg2-c61657-t2gzr. For the OpenShift Container Platform installation to succeed, all virtual machines comprising the cluster must be located in a single data center folder.
10. Click Save Configuration. This updates the cloud-provider-config file in the openshift-config namespace, and starts the configuration process.
11. Reopen the vSphere connection configuration wizard and expand the Monitored operators panel. Check that the status of the operators is either Progressing or Healthy.

1. Check that the configuration process completed successfully:

   1. In the Administrator perspective, navigate to Home > Overview.
   2. Under Status click Operators. Wait for all operator statuses to change from Progressing to All succeeded. A Failed status indicates that the configuration failed.
   3. Under Status, click Control Plane. Wait for the response rate of all Control Pane components to return to 100%. A Failed control plane component indicates that the configuration failed.

   A failure indicates that at least one of the connection settings is incorrect. Change the settings in the vSphere connection configuration wizard and save the configuration again.

2. Check that you are able to bind PersistentVolumeClaims objects by performing the following steps:

   1. Create a StorageClass object using the following YAML:

      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
       name: vsphere-sc
      provisioner: kubernetes.io/vsphere-volume
      parameters:
       datastore: YOURVCENTERDATASTORE
       diskformat: thin
      reclaimPolicy: Delete
      volumeBindingMode: Immediate

   2. Create a PersistentVolumeClaims object using the following YAML:

      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
       name: test-pvc
       namespace: openshift-config
       annotations:
         volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
       finalizers:
         - kubernetes.io/pvc-protection
      spec:
       accessModes:
         - ReadWriteOnce
       resources:
         requests:
          storage: 10Gi
       storageClassName: vsphere-sc
       volumeMode: Filesystem

   For instructions, see Dynamic provisioning in the OpenShift Container Platform documentation. To troubleshoot a PersistentVolumeClaims object, navigate to Storage → PersistentVolumeClaims in the Administrator perspective of the OpenShift Container Platform web console.