Deploying installer-provisioned clusters on bare metal

Procedure

1. Boot the node and enter the BIOS menu.
2. Set the node’s boot mode to UEFI Enabled.
3. Enable Secure Boot.

Procedure

1. Log in to the provisioner node via ssh.

2. Create a non-root user (kni) and provide that user with sudo privileges:

   # useradd kni

   # passwd kni

   # echo "kni ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/kni

   # chmod 0440 /etc/sudoers.d/kni

3. Create an ssh key for the new user:

   # su - kni -c "ssh-keygen -t ed25519 -f /home/kni/.ssh/id_rsa -N ''"

4. Log in as the new user on the provisioner node:

   # su - kni

5. Use Red Hat Subscription Manager to register the provisioner node:

   $ sudo subscription-manager register --username=<user> --password=<pass> --auto-attach
   $ sudo subscription-manager repos --enable=rhel-8-for-<architecture>-appstream-rpms --enable=rhel-8-for-<architecture>-baseos-rpms

   Note

   For more information about Red Hat Subscription Manager, see Using and Configuring Red Hat Subscription Manager.

6. Install the following packages:

   $ sudo dnf install -y libvirt qemu-kvm mkisofs python3-devel jq ipmitool

7. Modify the user to add the libvirt group to the newly created user:

   $ sudo usermod --append --groups libvirt <user>

8. Restart firewalld and enable the http service:

   $ sudo systemctl start firewalld

   $ sudo firewall-cmd --zone=public --add-service=http --permanent

   $ sudo firewall-cmd --reload

9. Start and enable the libvirtd service:

   $ sudo systemctl enable libvirtd --now

10. Create the default storage pool and start it:

    $ sudo virsh pool-define-as --name default --type dir --target /var/lib/libvirt/images

    $ sudo virsh pool-start default

    $ sudo virsh pool-autostart default

11. Create a pull-secret.txt file:

    $ vim pull-secret.txt

    In a web browser, navigate to Install OpenShift on Bare Metal with installer-provisioned infrastructure. Click Copy pull secret. Paste the contents into the pull-secret.txt file and save the contents in the kni user’s home directory.

Procedure

1. Log in to the node by using the ssh command.

2. View the NTP servers available to the node by running the following command:

   $ chronyc sources

   Example output

   MS Name/IP address         Stratum Poll Reach LastRx Last sample
   ===============================================================================
   ^+ time.cloudflare.com           3  10   377   187   -209us[ -209us] +/-   32ms
   ^+ t1.time.ir2.yahoo.com         2  10   377   185  -4382us[-4382us] +/-   23ms
   ^+ time.cloudflare.com           3  10   377   198   -996us[-1220us] +/-   33ms
   ^* brenbox.westnet.ie            1  10   377   193  -9538us[-9761us] +/-   24ms

3. Use the ping command to ensure that the node can access an NTP server, for example:

   $ ping time.cloudflare.com

   Example output

   PING time.cloudflare.com (162.159.200.123) 56(84) bytes of data.
   64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=1 ttl=54 time=32.3 ms
   64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=2 ttl=54 time=30.9 ms
   64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=3 ttl=54 time=36.7 ms
   ...

Procedure

1. Export the bare-metal network NIC name by running the following command:

   $ export PUB_CONN=<baremetal_nic_name>

2. Configure the bare-metal network:

   Note

   The SSH connection might disconnect after executing these steps.

   1. For a network using DHCP, run the following command:

      $ sudo nohup bash -c "
          nmcli con down \"$PUB_CONN\"
          nmcli con delete \"$PUB_CONN\"
          # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
          nmcli con down \"System $PUB_CONN\"
          nmcli con delete \"System $PUB_CONN\"
          nmcli connection add ifname baremetal type bridge <con_name> baremetal bridge.stp no 1
          nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
          pkill dhclient;dhclient baremetal
      "

      1

      Replace <con_name> with the connection name.

   2. For a network using static IP addressing and no DHCP network, run the following command:

      $ sudo nohup bash -c "
          nmcli con down \"$PUB_CONN\"
          nmcli con delete \"$PUB_CONN\"
          # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
          nmcli con down \"System $PUB_CONN\"
          nmcli con delete \"System $PUB_CONN\"
          nmcli connection add ifname baremetal type bridge con-name baremetal bridge.stp no ipv4.method manual ipv4.addr "x.x.x.x/yy" ipv4.gateway "a.a.a.a" ipv4.dns "b.b.b.b" 1
          nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
          nmcli con up baremetal
      "

      1

      Replace <con_name> with the connection name. Replace x.x.x.x/yy with the IP address and CIDR for the network. Replace a.a.a.a with the network gateway. Replace b.b.b.b with the IP address of the DNS server.

3. Optional: If you are deploying with a provisioning network, export the provisioning network NIC name by running the following command:

   $ export PROV_CONN=<prov_nic_name>

4. Optional: If you are deploying with a provisioning network, configure the provisioning network by running the following command:

   $ sudo nohup bash -c "
       nmcli con down \"$PROV_CONN\"
       nmcli con delete \"$PROV_CONN\"
       nmcli connection add ifname provisioning type bridge con-name provisioning
       nmcli con add type bridge-slave ifname \"$PROV_CONN\" master provisioning
       nmcli connection modify provisioning ipv6.addresses fd00:1101::1/64 ipv6.method manual
       nmcli con down provisioning
       nmcli con up provisioning
   "

   Note

   The SSH connection might disconnect after executing these steps.

   The IPv6 address can be any address that is not routable through the bare-metal network.

   Ensure that UEFI is enabled and UEFI PXE settings are set to the IPv6 protocol when using IPv6 addressing.

5. Optional: If you are deploying with a provisioning network, configure the IPv4 address on the provisioning network connection by running the following command:

   $ nmcli connection modify provisioning ipv4.addresses 172.22.0.254/24 ipv4.method manual

6. SSH back into the provisioner node (if required) by running the following command:

   # ssh kni@provisioner.<cluster-name>.<domain>

7. Verify that the connection bridges have been properly created by running the following command:

   $ sudo nmcli con show

   Example output

   NAME               UUID                                  TYPE      DEVICE
   baremetal          4d5133a5-8351-4bb9-bfd4-3af264801530  bridge    baremetal
   provisioning       43942805-017f-4d7d-a2c2-7cb3324482ed  bridge    provisioning
   virbr0             d9bca40f-eee1-410b-8879-a2d4bb0465e7  bridge    virbr0
   bridge-slave-eno1  76a8ed50-c7e5-4999-b4f6-6d9014dd0812  ethernet  eno1
   bridge-slave-eno2  f31c3353-54b7-48de-893a-02d2b34c4736  ethernet  eno2

Procedure

1. Configure the first subnet to communicate with the second subnet:

   1. Log in as root to a control plane node by running the following command:

      $ sudo su -

   2. Get the name of the network interface:

      # nmcli dev status

   3. Add a route to the second subnet (192.168.0.0) via the gateway: s+

1. Apply the changes:

   # nmcli connection up <interface_name>

   Replace <interface_name> with the interface name.

2. Verify the routing table to ensure the route has been added successfully:

   # ip route

3. Repeat the previous steps for each control plane node in the first subnet.

   Note

   Adjust the commands to match your actual interface names and gateway.

   1. Configure the second subnet to communicate with the first subnet:

4. Log in as root to a remote worker node:

   $ sudo su -

5. Get the name of the network interface:

   # nmcli dev status

6. Add a route to the first subnet (10.0.0.0) via the gateway:

   # nmcli connection modify <interface_name> +ipv4.routes "10.0.0.0/24 via <gateway>"

   Replace <interface_name> with the interface name. Replace <gateway> with the IP address of the actual gateway.

   Example

   # nmcli connection modify eth0 +ipv4.routes "10.0.0.0/24 via 10.0.0.1"

7. Apply the changes:

   # nmcli connection up <interface_name>

   Replace <interface_name> with the interface name.

8. Verify the routing table to ensure the route has been added successfully:

   # ip route

9. Repeat the previous steps for each worker node in the second subnet.

   Note

   Adjust the commands to match your actual interface names and gateway.

   1. Once you have configured the networks, test the connectivity to ensure the remote worker nodes can reach the control plane nodes and the control plane nodes can reach the remote worker nodes.

10. From the control plane nodes in the first subnet, ping a remote worker node in the second subnet:

    $ ping <remote_worker_node_ip_address>

    If the ping is successful, it means the control plane nodes in the first subnet can reach the remote worker nodes in the second subnet. If you don’t receive a response, review the network configurations and repeat the procedure for the node.

11. From the remote worker nodes in the second subnet, ping a control plane node in the first subnet:

    $ ping <control_plane_node_ip_address>

    If the ping is successful, it means the remote worker nodes in the second subnet can reach the control plane in the first subnet. If you don’t receive a response, review the network configurations and repeat the procedure for the node.

Procedure

1. Set the environment variables:

   $ export cmd=openshift-baremetal-install

   $ export pullsecret_file=~/pull-secret.txt

   $ export extract_dir=$(pwd)

2. Get the oc binary:

   $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux.tar.gz | tar zxvf - oc

3. Extract the installer:

   $ sudo cp oc /usr/local/bin

   $ oc adm release extract --registry-config "${pullsecret_file}" --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}

   $ sudo cp openshift-baremetal-install /usr/local/bin

Procedure

1. Install podman:

   $ sudo dnf install -y podman

2. Open firewall port 8080 to be used for RHCOS image caching:

   $ sudo firewall-cmd --add-port=8080/tcp --zone=public --permanent

   $ sudo firewall-cmd --reload

3. Create a directory to store the bootstraposimage:

   $ mkdir /home/kni/rhcos_image_cache

4. Set the appropriate SELinux context for the newly created directory:

   $ sudo semanage fcontext -a -t httpd_sys_content_t "/home/kni/rhcos_image_cache(/.*)?"

   $ sudo restorecon -Rv /home/kni/rhcos_image_cache/

5. Get the URI for the RHCOS image that the installation program will deploy on the bootstrap VM:

   $ export RHCOS_QEMU_URI=$(/usr/local/bin/openshift-baremetal-install coreos print-stream-json | jq -r --arg ARCH "$(arch)" '.architectures[$ARCH].artifacts.qemu.formats["qcow2.gz"].disk.location')

6. Get the name of the image that the installation program will deploy on the bootstrap VM:

   $ export RHCOS_QEMU_NAME=${RHCOS_QEMU_URI##*/}

7. Get the SHA hash for the RHCOS image that will be deployed on the bootstrap VM:

   $ export RHCOS_QEMU_UNCOMPRESSED_SHA256=$(/usr/local/bin/openshift-baremetal-install coreos print-stream-json | jq -r --arg ARCH "$(arch)" '.architectures[$ARCH].artifacts.qemu.formats["qcow2.gz"].disk["uncompressed-sha256"]')

8. Download the image and place it in the /home/kni/rhcos_image_cache directory:

   $ curl -L ${RHCOS_QEMU_URI} -o /home/kni/rhcos_image_cache/${RHCOS_QEMU_NAME}

9. Confirm SELinux type is of httpd_sys_content_t for the new file:

   $ ls -Z /home/kni/rhcos_image_cache

10. Create the pod:

    $ podman run -d --name rhcos_image_cache \1
    -v /home/kni/rhcos_image_cache:/var/www/html \
    -p 8080:8080/tcp \
    registry.access.redhat.com/ubi9/httpd-24

    1

    Creates a caching webserver with the name rhcos_image_cache. This pod serves the bootstrapOSImage image in the install-config.yaml file for deployment.

11. Generate the bootstrapOSImage configuration:

    $ export BAREMETAL_IP=$(ip addr show dev baremetal | awk '/inet /{print $2}' | cut -d"/" -f1)

    $ export BOOTSTRAP_OS_IMAGE="http://${BAREMETAL_IP}:8080/${RHCOS_QEMU_NAME}?sha256=${RHCOS_QEMU_UNCOMPRESSED_SHA256}"

    $ echo "    bootstrapOSImage=${BOOTSTRAP_OS_IMAGE}"

12. Add the required configuration to the install-config.yaml file under platform.baremetal:

    platform:
      baremetal:
        bootstrapOSImage: <bootstrap_os_image>  1

    1

    Replace <bootstrap_os_image> with the value of $BOOTSTRAP_OS_IMAGE.

    See the "Configuring the install-config.yaml file" section for additional details.

Procedure

1. Create the ignition configuration files:

   $ ./openshift-baremetal-install --dir <cluster_configs> create ignition-configs

   Replace <cluster_configs> with the path to your cluster configuration files.

2. Create the bootstrap_config.sh file:

   #!/bin/bash
   
   BOOTSTRAP_CONFIG="[connection]
   type=ethernet
   interface-name=ens3
   [ethernet]
   [ipv4]
   method=manual
   addresses=<ip_address>/<cidr>
   gateway=<gateway_ip_address>
   dns=<dns_ip_address>"
   
   cat <<_EOF_ > bootstrap_network_config.ign
   {
     "path": "/etc/NetworkManager/system-connections/ens3.nmconnection",
     "mode": 384,
     "contents": {
       "source": "data:text/plain;charset=utf-8;base64,$(echo "${BOOTSTRAP_CONFIG}" | base64 -w 0)"
     }
   }
   _EOF_
   
   mv <cluster_configs>/bootstrap.ign <cluster_configs>/bootstrap.ign.orig
   
   jq '.storage.files += $input' <cluster_configs>/bootstrap.ign.orig --slurpfile input bootstrap_network_config.ign > <cluster_configs>/bootstrap.ign

   Replace <ip_address> and <cidr> with the IP address and CIDR of the address range. Replace <gateway_ip_address> with the IP address of the gateway on the baremetal network. Replace <dns_ip_address> with the IP address of the DNS server on the baremetal network. Replace <cluster_configs> with the path to your cluster configuration files.

3. Make the bootstrap_config.sh file executable:

   $ chmod 755 bootstrap_config.sh

4. Run the bootstrap_config.sh script to create the bootstrap_network_config.ign file:

   $ ./bootstrap_config.sh

Procedure

1. Check the network interface configuration on the node.
2. Turn off the DHCP server and reboot the OpenShift Container Platform node and ensure that the network configuration works properly.

Procedure

1. Remove or reformat the disks for the bootstrap, control plane node, and worker nodes. If you are working in a hypervisor environment, you must add any disks you removed.

2. Delete the artifacts that the previous installation generated:

   $ cd ; /bin/rm -rf auth/ bootstrap.ign master.ign worker.ign metadata.json \
   .openshift_install.log .openshift_install_state.json

3. Generate new manifests and Ignition config files. See “Creating the Kubernetes manifest and Ignition config files" for more information.
4. Upload the new bootstrap, control plane, and compute node Ignition config files that the installation program created to your HTTP server. This will overwrite the previous Ignition files.

Procedure

1. Create a Butane config, 99-master-chrony-conf-override.bu, including the contents of the chrony.conf file for the control plane nodes.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   Butane config example

   variant: openshift
   version: 4.12.0
   metadata:
     name: 99-master-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: master
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # Use public servers from the pool.ntp.org project.
             # Please consider joining the pool (https://www.pool.ntp.org/join.html).
   
             # The Machine Config Operator manages this file
             server openshift-master-0.<cluster-name>.<domain> iburst 1
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony
   
             # Configure the control plane nodes to serve as local NTP servers
             # for all worker nodes, even if they are not in sync with an
             # upstream NTP server.
   
             # Allow NTP client access from the local network.
             allow all
             # Serve time even if not synchronized to a time source.
             local stratum 3 orphan

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

2. Use Butane to generate a MachineConfig object file, 99-master-chrony-conf-override.yaml, containing the configuration to be delivered to the control plane nodes:

   $ butane 99-master-chrony-conf-override.bu -o 99-master-chrony-conf-override.yaml

3. Create a Butane config, 99-worker-chrony-conf-override.bu, including the contents of the chrony.conf file for the worker nodes that references the NTP servers on the control plane nodes.

   Butane config example

   variant: openshift
   version: 4.12.0
   metadata:
     name: 99-worker-chrony-conf-override
     labels:
       machineconfiguration.openshift.io/role: worker
   storage:
     files:
       - path: /etc/chrony.conf
         mode: 0644
         overwrite: true
         contents:
           inline: |
             # The Machine Config Operator manages this file.
             server openshift-master-0.<cluster-name>.<domain> iburst 1
             server openshift-master-1.<cluster-name>.<domain> iburst
             server openshift-master-2.<cluster-name>.<domain> iburst
   
             stratumweight 0
             driftfile /var/lib/chrony/drift
             rtcsync
             makestep 10 3
             bindcmdaddress 127.0.0.1
             bindcmdaddress ::1
             keyfile /etc/chrony.keys
             commandkey 1
             generatecommandkey
             noclientlog
             logchange 0.5
             logdir /var/log/chrony

   1

   You must replace <cluster-name> with the name of the cluster and replace <domain> with the fully qualified domain name.

4. Use Butane to generate a MachineConfig object file, 99-worker-chrony-conf-override.yaml, containing the configuration to be delivered to the worker nodes:

   $ butane 99-worker-chrony-conf-override.bu -o 99-worker-chrony-conf-override.yaml

5. Apply the 99-master-chrony-conf-override.yaml policy to the control plane nodes.

   $ oc apply -f 99-master-chrony-conf-override.yaml

   Example output

   machineconfig.machineconfiguration.openshift.io/99-master-chrony-conf-override created

6. Apply the 99-worker-chrony-conf-override.yaml policy to the worker nodes.

   $ oc apply -f 99-worker-chrony-conf-override.yaml

   Example output

   machineconfig.machineconfiguration.openshift.io/99-worker-chrony-conf-override created

7. Check the status of the applied NTP settings.

   $ oc describe machineconfigpool

Procedure

1. When setting the provisioningInterface setting, first identify the provisioning interface name for the cluster nodes. For example, eth0 or eno1.
2. Enable the Preboot eXecution Environment (PXE) on the provisioning network interface of the cluster nodes.

3. Retrieve the current state of the provisioning network and save it to a provisioning custom resource (CR) file:

   $ oc get provisioning -o yaml > enable-provisioning-nw.yaml

4. Modify the provisioning CR file:

   $ vim ~/enable-provisioning-nw.yaml

   Scroll down to the provisioningNetwork configuration setting and change it from Disabled to Managed. Then, add the provisioningIP, provisioningNetworkCIDR, provisioningDHCPRange, provisioningInterface, and watchAllNameSpaces configuration settings after the provisioningNetwork setting. Provide appropriate values for each setting.

   apiVersion: v1
   items:
   - apiVersion: metal3.io/v1alpha1
     kind: Provisioning
     metadata:
       name: provisioning-configuration
     spec:
       provisioningNetwork: 1
       provisioningIP: 2
       provisioningNetworkCIDR: 3
       provisioningDHCPRange: 4
       provisioningInterface: 5
       watchAllNameSpaces: 6

   1

   The provisioningNetwork is one of Managed, Unmanaged, or Disabled. When set to Managed, Metal3 manages the provisioning network and the CBO deploys the Metal3 pod with a configured DHCP server. When set to Unmanaged, the system administrator configures the DHCP server manually.

   2

   The provisioningIP is the static IP address that the DHCP server and ironic use to provision the network. This static IP address must be within the provisioning subnet, and outside of the DHCP range. If you configure this setting, it must have a valid IP address even if the provisioning network is Disabled. The static IP address is bound to the metal3 pod. If the metal3 pod fails and moves to another server, the static IP address also moves to the new server.

   3

   The Classless Inter-Domain Routing (CIDR) address. If you configure this setting, it must have a valid CIDR address even if the provisioning network is Disabled. For example: 192.168.0.1/24.

   4

   The DHCP range. This setting is only applicable to a Managed provisioning network. Omit this configuration setting if the provisioning network is Disabled. For example: 192.168.0.64, 192.168.0.253.

   5

   The NIC name for the provisioning interface on cluster nodes. The provisioningInterface setting is only applicable to Managed and Unmanaged provisioning networks. Omit the provisioningInterface configuration setting if the provisioning network is Disabled. Omit the provisioningInterface configuration setting to use the bootMACAddress configuration setting instead.

   6

   Set this setting to true if you want metal3 to watch namespaces other than the default openshift-machine-api namespace. The default value is false.

5. Save the changes to the provisioning CR file.

6. Apply the provisioning CR file to the cluster:

   $ oc apply -f enable-provisioning-nw.yaml

Procedure

1. Get the oc binary:

   $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux-$VERSION.tar.gz | tar zxvf - oc

   $ sudo cp oc /usr/local/bin

2. Power off the bare metal node by using the baseboard management controller (BMC), and ensure it is off.

3. Retrieve the user name and password of the bare metal node’s baseboard management controller. Then, create base64 strings from the user name and password:

   $ echo -ne "root" | base64

   $ echo -ne "password" | base64

4. Create a configuration file for the bare metal node. Depending on whether you are using a static configuration or a DHCP server, use one of the following example bmh.yaml files, replacing values in the YAML to match your environment:

   $ vim bmh.yaml

   • Static configuration bmh.yaml:

     ---
     apiVersion: v1 1
     kind: Secret
     metadata:
      name: openshift-worker-<num>-network-config-secret 2
      namespace: openshift-machine-api
     type: Opaque
     stringData:
      nmstate: | 3
       interfaces: 4
       - name: <nic1_name> 5
         type: ethernet
         state: up
         ipv4:
           address:
           - ip: <ip_address> 6
             prefix-length: 24
           enabled: true
       dns-resolver:
         config:
           server:
           - <dns_ip_address> 7
       routes:
         config:
         - destination: 0.0.0.0/0
           next-hop-address: <next_hop_ip_address> 8
           next-hop-interface: <next_hop_nic1_name> 9
     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: openshift-worker-<num>-bmc-secret 10
       namespace: openshift-machine-api
     type: Opaque
     data:
       username: <base64_of_uid> 11
       password: <base64_of_pwd> 12
     ---
     apiVersion: metal3.io/v1alpha1
     kind: BareMetalHost
     metadata:
       name: openshift-worker-<num> 13
       namespace: openshift-machine-api
     spec:
       online: True
       bootMACAddress: <nic1_mac_address> 14
       bmc:
         address: <protocol>://<bmc_url> 15
         credentialsName: openshift-worker-<num>-bmc-secret 16
         disableCertificateVerification: True 17
         username: <bmc_username> 18
         password: <bmc_password> 19
       rootDeviceHints:
         deviceName: <root_device_hint> 20
       preprovisioningNetworkDataName: openshift-worker-<num>-network-config-secret 21

     1

     To configure the network interface for a newly created node, specify the name of the secret that contains the network configuration. Follow the nmstate syntax to define the network configuration for your node. See "Optional: Configuring host network interfaces in the install-config.yaml file" for details on configuring NMState syntax.

     2 10 13 16

     Replace <num> for the worker number of the bare metal node in the name fields, the credentialsName field, and the preprovisioningNetworkDataName field.

     3

     Add the NMState YAML syntax to configure the host interfaces.

     4

     Optional: If you have configured the network interface with nmstate, and you want to disable an interface, set state: up with the IP addresses set to enabled: false as shown:

     ---
        interfaces:
        - name: <nic_name>
          type: ethernet
          state: up
          ipv4:
            enabled: false
          ipv6:
            enabled: false

     5 6 7 8 9

     Replace <nic1_name>, <ip_address>, <dns_ip_address>, <next_hop_ip_address> and <next_hop_nic1_name> with appropriate values.

     11 12

     Replace <base64_of_uid> and <base64_of_pwd> with the base64 string of the user name and password.

     14

     Replace <nic1_mac_address> with the MAC address of the bare metal node’s first NIC. See the "BMC addressing" section for additional BMC configuration options.

     15

     Replace <protocol> with the BMC protocol, such as IPMI, RedFish, or others. Replace <bmc_url> with the URL of the bare metal node’s baseboard management controller.

     17

     To skip certificate validation, set disableCertificateVerification to true.

     18 19

     Replace <bmc_username> and <bmc_password> with the string of the BMC user name and password.

     20

     Optional: Replace <root_device_hint> with a device path if you specify a root device hint.

     21

     Optional: If you have configured the network interface for the newly created node, provide the network configuration secret name in the preprovisioningNetworkDataName of the BareMetalHost CR.

   • DHCP configuration bmh.yaml:

     ---
     apiVersion: v1
     kind: Secret
     metadata:
       name: openshift-worker-<num>-bmc-secret 1
       namespace: openshift-machine-api
     type: Opaque
     data:
       username: <base64_of_uid> 2
       password: <base64_of_pwd> 3
     ---
     apiVersion: metal3.io/v1alpha1
     kind: BareMetalHost
     metadata:
       name: openshift-worker-<num> 4
       namespace: openshift-machine-api
     spec:
       online: True
       bootMACAddress: <nic1_mac_address> 5
       bmc:
         address: <protocol>://<bmc_url> 6
         credentialsName: openshift-worker-<num>-bmc-secret 7
         disableCertificateVerification: True 8
         username: <bmc_username> 9
         password: <bmc_password> 10
       rootDeviceHints:
         deviceName: <root_device_hint> 11
       preprovisioningNetworkDataName: openshift-worker-<num>-network-config-secret 12

     1 4 7

     Replace <num> for the worker number of the bare metal node in the name fields, the credentialsName field, and the preprovisioningNetworkDataName field.

     2 3

     Replace <base64_of_uid> and <base64_of_pwd> with the base64 string of the user name and password.

     5

     Replace <nic1_mac_address> with the MAC address of the bare metal node’s first NIC. See the "BMC addressing" section for additional BMC configuration options.

     6

     Replace <protocol> with the BMC protocol, such as IPMI, RedFish, or others. Replace <bmc_url> with the URL of the bare metal node’s baseboard management controller.

     8

     To skip certificate validation, set disableCertificateVerification to true.

     9 10

     Replace <bmc_username> and <bmc_password> with the string of the BMC user name and password.

     11

     Optional: Replace <root_device_hint> with a device path if you specify a root device hint.

     12

     Optional: If you have configured the network interface for the newly created node, provide the network configuration secret name in the preprovisioningNetworkDataName of the BareMetalHost CR.

   Note

   If the MAC address of an existing bare metal node matches the MAC address of a bare metal host that you are attempting to provision, then the Ironic installation will fail. If the host enrollment, inspection, cleaning, or other Ironic steps fail, the Bare Metal Operator retries the installation continuously. See "Diagnosing a host duplicate MAC address" for more information.

5. Create the bare metal node:

   $ oc -n openshift-machine-api create -f bmh.yaml

   Example output

   secret/openshift-worker-<num>-network-config-secret created
   secret/openshift-worker-<num>-bmc-secret created
   baremetalhost.metal3.io/openshift-worker-<num> created

   Where <num> will be the worker number.

6. Power up and inspect the bare metal node:

   $ oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number.

   Example output

   NAME                    STATE       CONSUMER   ONLINE   ERROR
   openshift-worker-<num>  available              true

   Note

   To allow the worker node to join the cluster, scale the machineset object to the number of the BareMetalHost objects. You can scale nodes either manually or automatically. To scale nodes automatically, use the metal3.io/autoscale-to-hosts annotation for machineset.

Procedure

1. Ensure that the Bare Metal Operator is available:

   $ oc get clusteroperator baremetal

   Example output

   NAME        VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
   baremetal   4.12.0   True        False         False      3d15h

2. Remove the old BareMetalHost and Machine objects:

   $ oc delete bmh -n openshift-machine-api <host_name>
   $ oc delete machine -n openshift-machine-api <machine_name>

   Replace <host_name> with the name of the host and <machine_name> with the name of the machine. The machine name appears under the CONSUMER field.

   After you remove the BareMetalHost and Machine objects, then the machine controller automatically deletes the Node object.

3. Create the new BareMetalHost object and the secret to store the BMC credentials:

   $ cat <<EOF | oc apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: control-plane-<num>-bmc-secret 1
     namespace: openshift-machine-api
   data:
     username: <base64_of_uid> 2
     password: <base64_of_pwd> 3
   type: Opaque
   ---
   apiVersion: metal3.io/v1alpha1
   kind: BareMetalHost
   metadata:
     name: control-plane-<num> 4
     namespace: openshift-machine-api
   spec:
     automatedCleaningMode: disabled
     bmc:
       address: <protocol>://<bmc_ip> 5
       credentialsName: control-plane-<num>-bmc-secret 6
     bootMACAddress: <NIC1_mac_address> 7
     bootMode: UEFI
     externallyProvisioned: false
     online: true
   EOF

   1 4 6

   Replace <num> for the control plane number of the bare metal node in the name fields and the credentialsName field.

   2

   Replace <base64_of_uid> with the base64 string of the user name.

   3

   Replace <base64_of_pwd> with the base64 string of the password.

   5

   Replace <protocol> with the BMC protocol, such as redfish, redfish-virtualmedia, idrac-virtualmedia, or others. Replace <bmc_ip> with the IP address of the bare metal node’s baseboard management controller. For additional BMC configuration options, see "BMC addressing" in the Additional resources section.

   7

   Replace <NIC1_mac_address> with the MAC address of the bare metal node’s first NIC.

   After the inspection is complete, the BareMetalHost object is created and available to be provisioned.

4. View available BareMetalHost objects:

   $ oc get bmh -n openshift-machine-api

   Example output

   NAME                          STATE                    CONSUMER                   ONLINE   ERROR   AGE
   control-plane-1.example.com   available                control-plane-1            true             1h10m
   control-plane-2.example.com   externally provisioned   control-plane-2            true             4h53m
   control-plane-3.example.com   externally provisioned   control-plane-3            true             4h53m
   compute-1.example.com         provisioned              compute-1-ktmmx            true             4h53m
   compute-1.example.com         provisioned              compute-2-l2zmb            true             4h53m

   There are no MachineSet objects for control plane nodes, so you must create a Machine object instead. You can copy the providerSpec from another control plane Machine object.

5. Create a Machine object:

   $ cat <<EOF | oc apply -f -
   apiVersion: machine.openshift.io/v1beta1
   kind: Machine
   metadata:
     annotations:
       metal3.io/BareMetalHost: openshift-machine-api/control-plane-<num> 1
     labels:
       machine.openshift.io/cluster-api-cluster: control-plane-<num> 2
       machine.openshift.io/cluster-api-machine-role: master
       machine.openshift.io/cluster-api-machine-type: master
     name: control-plane-<num> 3
     namespace: openshift-machine-api
   spec:
     metadata: {}
     providerSpec:
       value:
         apiVersion: baremetal.cluster.k8s.io/v1alpha1
         customDeploy:
           method: install_coreos
         hostSelector: {}
         image:
           checksum: ""
           url: ""
         kind: BareMetalMachineProviderSpec
         metadata:
           creationTimestamp: null
         userData:
           name: master-user-data-managed
   EOF

   1 2 3

   Replace <num> for the control plane number of the bare metal node in the name, labels and annotations fields.

6. To view the BareMetalHost objects, run the following command:

   $ oc get bmh -A

   Example output

   NAME                          STATE                    CONSUMER                   ONLINE   ERROR   AGE
   control-plane-1.example.com   provisioned              control-plane-1            true             2h53m
   control-plane-2.example.com   externally provisioned   control-plane-2            true             5h53m
   control-plane-3.example.com   externally provisioned   control-plane-3            true             5h53m
   compute-1.example.com         provisioned              compute-1-ktmmx            true             5h53m
   compute-2.example.com         provisioned              compute-2-l2zmb            true             5h53m

7. After the RHCOS installation, verify that the BareMetalHost is added to the cluster:

   $ oc get nodes

   Example output

   NAME                           STATUS      ROLES     AGE   VERSION
   control-plane-1.example.com    available   master    4m2s  v1.18.2
   control-plane-2.example.com    available   master    141m  v1.18.2
   control-plane-3.example.com    available   master    141m  v1.18.2
   compute-1.example.com          available   worker    87m   v1.18.2
   compute-2.example.com          available   worker    87m   v1.18.2

   Note

   After replacement of the new control plane node, the etcd pod running in the new node is in crashloopback status. See "Replacing an unhealthy etcd member" in the Additional resources section for more information.

Procedure

1. Edit the provisioning custom resource (CR) to enable deploying with Virtual Media on the baremetal network:

   oc edit provisioning

     apiVersion: metal3.io/v1alpha1
     kind: Provisioning
     metadata:
       creationTimestamp: "2021-08-05T18:51:50Z"
       finalizers:
       - provisioning.metal3.io
       generation: 8
       name: provisioning-configuration
       resourceVersion: "551591"
       uid: f76e956f-24c6-4361-aa5b-feaf72c5b526
     spec:
       provisioningDHCPRange: 172.22.0.10,172.22.0.254
       provisioningIP: 172.22.0.3
       provisioningInterface: enp1s0
       provisioningNetwork: Managed
       provisioningNetworkCIDR: 172.22.0.0/24
       virtualMediaViaExternalNetwork: true 1
     status:
       generations:
       - group: apps
         hash: ""
         lastGeneration: 7
         name: metal3
         namespace: openshift-machine-api
         resource: deployments
       - group: apps
         hash: ""
         lastGeneration: 1
         name: metal3-image-cache
         namespace: openshift-machine-api
         resource: daemonsets
       observedGeneration: 8
       readyReplicas: 0

   1

   Add virtualMediaViaExternalNetwork: true to the provisioning CR.

2. If the image URL exists, edit the machineset to use the API VIP address. This step only applies to clusters installed in versions 4.9 or earlier.

   oc edit machineset

     apiVersion: machine.openshift.io/v1beta1
     kind: MachineSet
     metadata:
       creationTimestamp: "2021-08-05T18:51:52Z"
       generation: 11
       labels:
         machine.openshift.io/cluster-api-cluster: ostest-hwmdt
         machine.openshift.io/cluster-api-machine-role: worker
         machine.openshift.io/cluster-api-machine-type: worker
       name: ostest-hwmdt-worker-0
       namespace: openshift-machine-api
       resourceVersion: "551513"
       uid: fad1c6e0-b9da-4d4a-8d73-286f78788931
     spec:
       replicas: 2
       selector:
         matchLabels:
           machine.openshift.io/cluster-api-cluster: ostest-hwmdt
           machine.openshift.io/cluster-api-machineset: ostest-hwmdt-worker-0
       template:
         metadata:
           labels:
             machine.openshift.io/cluster-api-cluster: ostest-hwmdt
             machine.openshift.io/cluster-api-machine-role: worker
             machine.openshift.io/cluster-api-machine-type: worker
             machine.openshift.io/cluster-api-machineset: ostest-hwmdt-worker-0
         spec:
           metadata: {}
           providerSpec:
             value:
               apiVersion: baremetal.cluster.k8s.io/v1alpha1
               hostSelector: {}
               image:
                 checksum: http:/172.22.0.3:6181/images/rhcos-<version>.<architecture>.qcow2.<md5sum> 1
                 url: http://172.22.0.3:6181/images/rhcos-<version>.<architecture>.qcow2 2
               kind: BareMetalMachineProviderSpec
               metadata:
                 creationTimestamp: null
               userData:
                 name: worker-user-data
     status:
       availableReplicas: 2
       fullyLabeledReplicas: 2
       observedGeneration: 11
       readyReplicas: 2
       replicas: 2

   1

   Edit the checksum URL to use the API VIP address.

   2

   Edit the url URL to use the API VIP address.

1. Get the bare-metal hosts running in the openshift-machine-api namespace:

   $ oc get bmh -n openshift-machine-api

   Example output

   NAME                 STATUS   PROVISIONING STATUS      CONSUMER
   openshift-master-0   OK       externally provisioned   openshift-zpwpq-master-0
   openshift-master-1   OK       externally provisioned   openshift-zpwpq-master-1
   openshift-master-2   OK       externally provisioned   openshift-zpwpq-master-2
   openshift-worker-0   OK       provisioned              openshift-zpwpq-worker-0-lv84n
   openshift-worker-1   OK       provisioned              openshift-zpwpq-worker-0-zd8lm
   openshift-worker-2   error    registering

2. To see more detailed information about the status of the failing host, run the following command replacing <bare_metal_host_name> with the name of the host:

   $ oc get -n openshift-machine-api bmh <bare_metal_host_name> -o yaml

   Example output

   ...
   status:
     errorCount: 12
     errorMessage: MAC address b4:96:91:1d:7c:20 conflicts with existing node openshift-worker-1
     errorType: registration error
   ...

Procedure

1. Ensure the STATE is available before provisioning the bare metal node.

   $  oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number.

   NAME              STATE     ONLINE ERROR  AGE
   openshift-worker  available true          34h

2. Get a count of the number of worker nodes.

   $ oc get nodes

   NAME                                                STATUS   ROLES           AGE     VERSION
   openshift-master-1.openshift.example.com            Ready    master          30h     v1.25.0
   openshift-master-2.openshift.example.com            Ready    master          30h     v1.25.0
   openshift-master-3.openshift.example.com            Ready    master          30h     v1.25.0
   openshift-worker-0.openshift.example.com            Ready    worker          30h     v1.25.0
   openshift-worker-1.openshift.example.com            Ready    worker          30h     v1.25.0

3. Get the compute machine set.

   $ oc get machinesets -n openshift-machine-api

   NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
   ...
   openshift-worker-0.example.com      1         1         1       1           55m
   openshift-worker-1.example.com      1         1         1       1           55m

4. Increase the number of worker nodes by one.

   $ oc scale --replicas=<num> machineset <machineset> -n openshift-machine-api

   Replace <num> with the new number of worker nodes. Replace <machineset> with the name of the compute machine set from the previous step.

5. Check the status of the bare metal node.

   $ oc -n openshift-machine-api get bmh openshift-worker-<num>

   Where <num> is the worker node number. The STATE changes from ready to provisioning.

   NAME                    STATE             CONSUMER                          ONLINE   ERROR
   openshift-worker-<num>  provisioning      openshift-worker-<num>-65tjz      true

   The provisioning status remains until the OpenShift Container Platform cluster provisions the node. This can take 30 minutes or more. After the node is provisioned, the state will change to provisioned.

   NAME                    STATE             CONSUMER                          ONLINE   ERROR
   openshift-worker-<num>  provisioned       openshift-worker-<num>-65tjz      true

6. After provisioning completes, ensure the bare metal node is ready.

   $ oc get nodes

   NAME                                          STATUS   ROLES   AGE     VERSION
   openshift-master-1.openshift.example.com      Ready    master  30h     v1.25.0
   openshift-master-2.openshift.example.com      Ready    master  30h     v1.25.0
   openshift-master-3.openshift.example.com      Ready    master  30h     v1.25.0
   openshift-worker-0.openshift.example.com      Ready    worker  30h     v1.25.0
   openshift-worker-1.openshift.example.com      Ready    worker  30h     v1.25.0
   openshift-worker-<num>.openshift.example.com  Ready    worker  3m27s   v1.25.0

   You can also check the kubelet.

   $ ssh openshift-worker-<num>

   [kni@openshift-worker-<num>]$ journalctl -fu kubelet

Procedure

1. Use the guidelines in YAML-tips.
2. Verify the YAML syntax is correct using syntax-check.

3. Verify the Red Hat Enterprise Linux CoreOS (RHCOS) QEMU images are properly defined and accessible via the URL provided in the install-config.yaml. For example:

   $ curl -s -o /dev/null -I -w "%{http_code}\n" http://webserver.example.com:8080/rhcos-44.81.202004250133-0-qemu.<architecture>.qcow2.gz?sha256=7d884b46ee54fe87bbc3893bf2aa99af3b2d31f2e19ab5529c60636fbd0f1ce7

   If the output is 200, there is a valid response from the webserver storing the bootstrap VM image.

Procedure

1. About 10 to 15 minutes after triggering the installation program, check to ensure the bootstrap VM is operational using the virsh command:

   $ sudo virsh list

    Id    Name                           State
    --------------------------------------------
    12    openshift-xf6fq-bootstrap      running

   Note

   The name of the bootstrap VM is always the cluster name followed by a random set of characters and ending in the word "bootstrap."

   If the bootstrap VM is not running after 10-15 minutes, troubleshoot why it is not running. Possible issues include:

2. Verify libvirtd is running on the system:

   $ systemctl status libvirtd

   ● libvirtd.service - Virtualization daemon
      Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
      Active: active (running) since Tue 2020-03-03 21:21:07 UTC; 3 weeks 5 days ago
        Docs: man:libvirtd(8)
              https://libvirt.org
    Main PID: 9850 (libvirtd)
       Tasks: 20 (limit: 32768)
      Memory: 74.8M
      CGroup: /system.slice/libvirtd.service
              ├─ 9850 /usr/sbin/libvirtd

   If the bootstrap VM is operational, log in to it.

3. Use the virsh console command to find the IP address of the bootstrap VM:

   $ sudo virsh console example.com

   Connected to domain example.com
   Escape character is ^]
   Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.3
   SSH host key: SHA256:BRWJktXZgQQRY5zjuAV0IKZ4WM7i4TiUyMVanqu9Pqg (ED25519)
   SSH host key: SHA256:7+iKGA7VtG5szmk2jB5gl/5EZ+SNcJ3a2g23o0lnIio (ECDSA)
   SSH host key: SHA256:DH5VWhvhvagOTaLsYiVNse9ca+ZSW/30OOMed8rIGOc (RSA)
   ens3:  fd35:919d:4042:2:c7ed:9a9f:a9ec:7
   ens4: 172.22.0.2 fe80::1d05:e52e:be5d:263f
   localhost login:

   Important

   When deploying an OpenShift Container Platform cluster without the provisioning network, you must use a public IP address and not a private IP address like 172.22.0.2.

4. After you obtain the IP address, log in to the bootstrap VM using the ssh command:

   Note

   In the console output of the previous step, you can use the IPv6 IP address provided by ens3 or the IPv4 IP provided by ens4.

   $ ssh core@172.22.0.2

Procedure

1. Check the network connectivity to the provisioning network.
2. Ensure PXE is enabled on the NIC for the provisioning network and PXE is disabled for all other NICs.

3. Verify that the install-config.yaml configuration file includes the rootDeviceHints parameter and boot MAC address for the NIC connected to the provisioning network. For example:

   control plane node settings

   bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC

   Worker node settings

   bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC

Procedure

1. Hostname Resolution: Check the cluster nodes to ensure they have a fully qualified domain name, and not just localhost.localdomain. For example:

   $ hostname

   If a hostname is not set, set the correct hostname. For example:

   $ hostnamectl set-hostname <hostname>

2. Incorrect Name Resolution: Ensure that each node has the correct name resolution in the DNS server using dig and nslookup. For example:

   $ dig api.<cluster_name>.example.com

   ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el8 <<>> api.<cluster_name>.example.com
   ;; global options: +cmd
   ;; Got answer:
   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37551
   ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
   
   ;; OPT PSEUDOSECTION:
   ; EDNS: version: 0, flags:; udp: 4096
   ; COOKIE: 866929d2f8e8563582af23f05ec44203d313e50948d43f60 (good)
   ;; QUESTION SECTION:
   ;api.<cluster_name>.example.com. IN A
   
   ;; ANSWER SECTION:
   api.<cluster_name>.example.com. 10800 IN	A 10.19.13.86
   
   ;; AUTHORITY SECTION:
   <cluster_name>.example.com. 10800 IN NS	<cluster_name>.example.com.
   
   ;; ADDITIONAL SECTION:
   <cluster_name>.example.com. 10800 IN A	10.19.14.247
   
   ;; Query time: 0 msec
   ;; SERVER: 10.19.14.247#53(10.19.14.247)
   ;; WHEN: Tue May 19 20:30:59 UTC 2020
   ;; MSG SIZE  rcvd: 140

   The output in the foregoing example indicates that the appropriate IP address for the api.<cluster_name>.example.com VIP is 10.19.13.86. This IP address should reside on the baremetal network.

Procedure

1. Add a DNS A/AAAA or CNAME record to internally identify the API load balancer. For example, when using dnsmasq, modify the dnsmasq.conf configuration file:

   $ sudo nano /etc/dnsmasq.conf

   address=/api-int.<cluster_name>.<base_domain>/<IP_address>
   address=/api-int.mycluster.example.com/192.168.1.10
   address=/api-int.mycluster.example.com/2001:0db8:85a3:0000:0000:8a2e:0370:7334

2. Add a DNS PTR record to internally identify the API load balancer. For example, when using dnsmasq, modify the dnsmasq.conf configuration file:

   $ sudo nano /etc/dnsmasq.conf

   ptr-record=<IP_address>.in-addr.arpa,api-int.<cluster_name>.<base_domain>
   ptr-record=10.1.168.192.in-addr.arpa,api-int.mycluster.example.com

3. Restart the DNS server. For example, when using dnsmasq, execute the following command:

   $ sudo systemctl restart dnsmasq

Procedure

1. Power off all bare metal nodes prior to installing the OpenShift Container Platform cluster:

   $ ipmitool -I lanplus -U <user> -P <password> -H <management_server_ip> power off

2. Remove all old bootstrap resources if any are left over from a previous deployment attempt:

   for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
   do
     sudo virsh destroy $i;
     sudo virsh undefine $i;
     sudo virsh vol-delete $i --pool $i;
     sudo virsh vol-delete $i.ign --pool $i;
     sudo virsh pool-destroy $i;
     sudo virsh pool-undefine $i;
   done

3. Remove the following from the clusterconfigs directory to prevent Terraform from failing:

   $ rm -rf ~/clusterconfigs/auth ~/clusterconfigs/terraform* ~/clusterconfigs/tls ~/clusterconfigs/metadata.json

Procedure

1. Check to ensure authentication is successful:

   $ /usr/local/bin/oc adm release mirror \
     -a pull-secret-update.json
     --from=$UPSTREAM_REPO \
     --to-release-image=$LOCAL_REG/$LOCAL_REPO:${VERSION} \
     --to=$LOCAL_REG/$LOCAL_REPO

   Note

   Example output of the variables used to mirror the install images:

   UPSTREAM_REPO=${RELEASE_IMAGE}
   LOCAL_REG=<registry_FQDN>:<registry_port>
   LOCAL_REPO='ocp4/openshift4'

   The values of RELEASE_IMAGE and VERSION were set during the Retrieving OpenShift Installer step of the Setting up the environment for an OpenShift installation section.

2. After mirroring the registry, confirm that you can access it in your disconnected environment:

   $ curl -k -u <user>:<password> https://registry.example.com:<registry_port>/v2/_catalog
   {"repositories":["<Repo_Name>"]}

Procedure

1. When the OpenShift Container Platform cluster nodes are installed appropriately, the following Ready state is seen within the STATUS column:

   $ oc get nodes

   NAME                   STATUS   ROLES           AGE  VERSION
   master-0.example.com   Ready    master,worker   4h   v1.25.0
   master-1.example.com   Ready    master,worker   4h   v1.25.0
   master-2.example.com   Ready    master,worker   4h   v1.25.0

2. Confirm the installer deployed all pods successfully. The following command removes any pods that are still running or have completed as part of the output.

   $ oc get pods --all-namespaces | grep -iv running | grep -iv complete