Chapter 5. Troubleshooting the installation

Procedure

1. Use the guidelines in YAML-tips.
2. Verify the YAML syntax is correct using syntax-check.

3. Verify the Red Hat Enterprise Linux CoreOS (RHCOS) QEMU images are properly defined and accessible via the URL provided in the install-config.yaml. For example:

   $ curl -s -o /dev/null -I -w "%{http_code}\n" http://webserver.example.com:8080/rhcos-44.81.202004250133-0-qemu.<architecture>.qcow2.gz?sha256=7d884b46ee54fe87bbc3893bf2aa99af3b2d31f2e19ab5529c60636fbd0f1ce7

   If the output is 200, there is a valid response from the webserver storing the bootstrap VM image.

Procedure

1. About 10 to 15 minutes after triggering the installation program, check to ensure the bootstrap VM is operational using the virsh command:

   $ sudo virsh list

    Id    Name                           State
    --------------------------------------------
    12    openshift-xf6fq-bootstrap      running

   Note

   The name of the bootstrap VM is always the cluster name followed by a random set of characters and ending in the word "bootstrap."

2. If the bootstrap VM is not running after 10-15 minutes, verify libvirtd is running on the system by executing the following command:

   $ systemctl status libvirtd

   ● libvirtd.service - Virtualization daemon
      Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
      Active: active (running) since Tue 2020-03-03 21:21:07 UTC; 3 weeks 5 days ago
        Docs: man:libvirtd(8)
              https://libvirt.org
    Main PID: 9850 (libvirtd)
       Tasks: 20 (limit: 32768)
      Memory: 74.8M
      CGroup: /system.slice/libvirtd.service
              ├─ 9850 /usr/sbin/libvirtd

   If the bootstrap VM is operational, log in to it.

3. Use the virsh console command to find the IP address of the bootstrap VM:

   $ sudo virsh console example.com

   Connected to domain example.com
   Escape character is ^]
   Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.3
   SSH host key: SHA256:BRWJktXZgQQRY5zjuAV0IKZ4WM7i4TiUyMVanqu9Pqg (ED25519)
   SSH host key: SHA256:7+iKGA7VtG5szmk2jB5gl/5EZ+SNcJ3a2g23o0lnIio (ECDSA)
   SSH host key: SHA256:DH5VWhvhvagOTaLsYiVNse9ca+ZSW/30OOMed8rIGOc (RSA)
   ens3:  fd35:919d:4042:2:c7ed:9a9f:a9ec:7
   ens4: 172.22.0.2 fe80::1d05:e52e:be5d:263f
   localhost login:

   Important

   When deploying an OpenShift Container Platform cluster without the provisioning network, you must use a public IP address and not a private IP address like 172.22.0.2.

4. After you obtain the IP address, log in to the bootstrap VM using the ssh command:

   Note

   In the console output of the previous step, you can use the IPv6 IP address provided by ens3 or the IPv4 IP provided by ens4.

   $ ssh core@172.22.0.2

Procedure

1. Ensure that etcd is running on each of the control plane nodes by running the following command:

   $ sudo crictl logs $(sudo crictl ps --pod=$(sudo crictl pods --name=etcd-member --quiet) --quiet)

2. If the previous command fails, ensure that Kubelet created the etcd pods by running the following command:

   $ sudo crictl pods --name=etcd-member

   If there are no pods, investigate etcd.

3. Check the cluster nodes to ensure they have a fully qualified domain name, and not just localhost.localdomain, by using the following command:

   $ hostname

   If a hostname is not set, set the correct hostname. For example:

   $ sudo hostnamectl set-hostname <hostname>

4. Ensure that each node has the correct name resolution in the DNS server using the dig command:

   $ dig api.<cluster_name>.example.com

   ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el8 <<>> api.<cluster_name>.example.com
   ;; global options: +cmd
   ;; Got answer:
   ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37551
   ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
   
   ;; OPT PSEUDOSECTION:
   ; EDNS: version: 0, flags:; udp: 4096
   ; COOKIE: 866929d2f8e8563582af23f05ec44203d313e50948d43f60 (good)
   ;; QUESTION SECTION:
   ;api.<cluster_name>.example.com. IN A
   
   ;; ANSWER SECTION:
   api.<cluster_name>.example.com. 10800 IN	A 10.19.13.86
   
   ;; AUTHORITY SECTION:
   <cluster_name>.example.com. 10800 IN NS	<cluster_name>.example.com.
   
   ;; ADDITIONAL SECTION:
   <cluster_name>.example.com. 10800 IN A	10.19.14.247
   
   ;; Query time: 0 msec
   ;; SERVER: 10.19.14.247#53(10.19.14.247)
   ;; WHEN: Tue May 19 20:30:59 UTC 2020
   ;; MSG SIZE  rcvd: 140

   The output in the foregoing example indicates that the appropriate IP address for the api.<cluster_name>.example.com VIP is 10.19.13.86. This IP address should reside on the baremetal network.

Procedure

1. Inspect the ClusterVersion object by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusterversion -o yaml

   Example output

   apiVersion: config.openshift.io/v1
   kind: ClusterVersion
   metadata:
     creationTimestamp: 2019-02-27T22:24:21Z
     generation: 1
     name: version
     resourceVersion: "19927"
     selfLink: /apis/config.openshift.io/v1/clusterversions/version
     uid: 6e0f4cf8-3ade-11e9-9034-0a923b47ded4
   spec:
     channel: stable-4.1
     clusterID: 5ec312f9-f729-429d-a454-61d4906896ca
   status:
     availableUpdates: null
     conditions:
     - lastTransitionTime: 2019-02-27T22:50:30Z
       message: Done applying 4.1.1
       status: "True"
       type: Available
     - lastTransitionTime: 2019-02-27T22:50:30Z
       status: "False"
       type: Failing
     - lastTransitionTime: 2019-02-27T22:50:30Z
       message: Cluster version is 4.1.1
       status: "False"
       type: Progressing
     - lastTransitionTime: 2019-02-27T22:24:31Z
       message: 'Unable to retrieve available updates: unknown version 4.1.1
       reason: RemoteFailed
       status: "False"
       type: RetrievedUpdates
     desired:
       image: registry.svc.ci.openshift.org/openshift/origin-release@sha256:91e6f754975963e7db1a9958075eb609ad226968623939d262d1cf45e9dbc39a
       version: 4.1.1
     history:
     - completionTime: 2019-02-27T22:50:30Z
       image: registry.svc.ci.openshift.org/openshift/origin-release@sha256:91e6f754975963e7db1a9958075eb609ad226968623939d262d1cf45e9dbc39a
       startedTime: 2019-02-27T22:24:31Z
       state: Completed
       version: 4.1.1
     observedGeneration: 1
     versionHash: Wa7as_ik1qE=

2. View the conditions by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusterversion version \
        -o=jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}'

   Some of most important conditions include Failing, Available and Progressing.

   Example output

   Available True Done applying 4.1.1
   Failing False
   Progressing False Cluster version is 4.0.0-0.alpha-2019-02-26-194020
   RetrievedUpdates False Unable to retrieve available updates: unknown version 4.1.1

3. Inspect the ClusterOperator object by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator

   The command returns the status of the cluster Operators.

   Example output

   NAME                                  VERSION   AVAILABLE   PROGRESSING   FAILING   SINCE
   cluster-baremetal-operator                      True        False         False     17m
   cluster-autoscaler                              True        False         False     17m
   cluster-storage-operator                        True        False         False     10m
   console                                         True        False         False     7m21s
   dns                                             True        False         False     31m
   image-registry                                  True        False         False     9m58s
   ingress                                         True        False         False     10m
   kube-apiserver                                  True        False         False     28m
   kube-controller-manager                         True        False         False     21m
   kube-scheduler                                  True        False         False     25m
   machine-api                                     True        False         False     17m
   machine-config                                  True        False         False     17m
   marketplace-operator                            True        False         False     10m
   monitoring                                      True        False         False     8m23s
   network                                         True        False         False     13m
   node-tuning                                     True        False         False     11m
   openshift-apiserver                             True        False         False     15m
   openshift-authentication                        True        False         False     20m
   openshift-cloud-credential-operator             True        False         False     18m
   openshift-controller-manager                    True        False         False     10m
   openshift-samples                               True        False         False     8m42s
   operator-lifecycle-manager                      True        False         False     17m
   service-ca                                      True        False         False     30m

4. Inspect individual cluster Operators by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator <operator> -oyaml 1

   1

   Replace <operator> with the name of a cluster Operator. This command is useful for identifying why an cluster Operator has not achieved the Available state or is in the Failed state.

   Example output

   apiVersion: config.openshift.io/v1
   kind: ClusterOperator
   metadata:
     creationTimestamp: 2019-02-27T22:47:04Z
     generation: 1
     name: monitoring
     resourceVersion: "24677"
     selfLink: /apis/config.openshift.io/v1/clusteroperators/monitoring
     uid: 9a6a5ef9-3ae1-11e9-bad4-0a97b6ba9358
   spec: {}
   status:
     conditions:
     - lastTransitionTime: 2019-02-27T22:49:10Z
       message: Successfully rolled out the stack.
       status: "True"
       type: Available
     - lastTransitionTime: 2019-02-27T22:49:10Z
       status: "False"
       type: Progressing
     - lastTransitionTime: 2019-02-27T22:49:10Z
       status: "False"
       type: Failing
     extension: null
     relatedObjects: null
     version: ""

5. To get the cluster Operator’s status condition, run the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator <operator> \
        -o=jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}'

   Replace <operator> with the name of one of the operators above.

   Example output

   Available True Successfully rolled out the stack
   Progressing False
   Failing False

6. To retrieve the list of objects owned by the cluster Operator, execute the following command:

   oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator kube-apiserver \
      -o=jsonpath='{.status.relatedObjects}'

   Example output

   [map[resource:kubeapiservers group:operator.openshift.io name:cluster] map[group: name:openshift-config resource:namespaces] map[group: name:openshift-config-managed resource:namespaces] map[group: name:openshift-kube-apiserver-operator resource:namespaces] map[group: name:openshift-kube-apiserver resource:namespaces]]

Procedure

1. Check if the console router is in the Available or Failing state by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator console -oyaml

   apiVersion: config.openshift.io/v1
   kind: ClusterOperator
   metadata:
     creationTimestamp: 2019-02-27T22:46:57Z
     generation: 1
     name: console
     resourceVersion: "19682"
     selfLink: /apis/config.openshift.io/v1/clusteroperators/console
     uid: 960364aa-3ae1-11e9-bad4-0a97b6ba9358
   spec: {}
   status:
     conditions:
     - lastTransitionTime: 2019-02-27T22:46:58Z
       status: "False"
       type: Failing
     - lastTransitionTime: 2019-02-27T22:50:12Z
       status: "False"
       type: Progressing
     - lastTransitionTime: 2019-02-27T22:50:12Z
       status: "True"
       type: Available
     - lastTransitionTime: 2019-02-27T22:46:57Z
       status: "True"
       type: Upgradeable
     extension: null
     relatedObjects:
     - group: operator.openshift.io
       name: cluster
       resource: consoles
     - group: config.openshift.io
       name: cluster
       resource: consoles
     - group: oauth.openshift.io
       name: console
       resource: oauthclients
     - group: ""
       name: openshift-console-operator
       resource: namespaces
     - group: ""
       name: openshift-console
       resource: namespaces
     versions: null

2. Manually retrieve the console URL by executing the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get route console -n openshift-console \
        -o=jsonpath='{.spec.host}' console-openshift-console.apps.adahiya-1.devcluster.openshift.com

Procedure

1. Retrieve the certificate from the cluster using the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get configmaps default-ingress-cert \
        -n openshift-config-managed -o=jsonpath='{.data.ca-bundle\.crt}'

   -----BEGIN CERTIFICATE-----
   MIIC/TCCAeWgAwIBAgIBATANBgkqhkiG9w0BAQsFADAuMSwwKgYDVQQDDCNjbHVz
   dGVyLWluZ3Jlc3Mtb3BlcmF0b3JAMTU1MTMwNzU4OTAeFw0xOTAyMjcyMjQ2Mjha
   Fw0yMTAyMjYyMjQ2MjlaMC4xLDAqBgNVBAMMI2NsdXN0ZXItaW5ncmVzcy1vcGVy
   YXRvckAxNTUxMzA3NTg5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
   uCA4fQ+2YXoXSUL4h/mcvJfrgpBfKBW5hfB8NcgXeCYiQPnCKblH1sEQnI3VC5Pk
   2OfNCF3PUlfm4i8CHC95a7nCkRjmJNg1gVrWCvS/ohLgnO0BvszSiRLxIpuo3C4S
   EVqqvxValHcbdAXWgZLQoYZXV7RMz8yZjl5CfhDaaItyBFj3GtIJkXgUwp/5sUfI
   LDXW8MM6AXfuG+kweLdLCMm3g8WLLfLBLvVBKB+4IhIH7ll0buOz04RKhnYN+Ebw
   tcvFi55vwuUCWMnGhWHGEQ8sWm/wLnNlOwsUz7S1/sW8nj87GFHzgkaVM9EOnoNI
   gKhMBK9ItNzjrP6dgiKBCQIDAQABoyYwJDAOBgNVHQ8BAf8EBAMCAqQwEgYDVR0T
   AQH/BAgwBgEB/wIBADANBgkqhkiG9w0BAQsFAAOCAQEAq+vi0sFKudaZ9aUQMMha
   CeWx9CZvZBblnAWT/61UdpZKpFi4eJ2d33lGcfKwHOi2NP/iSKQBebfG0iNLVVPz
   vwLbSG1i9R9GLdAbnHpPT9UG6fLaDIoKpnKiBfGENfxeiq5vTln2bAgivxrVlyiq
   +MdDXFAWb6V4u2xh6RChI7akNsS3oU9PZ9YOs5e8vJp2YAEphht05X0swA+X8V8T
   C278FFifpo0h3Q0Dbv8Rfn4UpBEtN4KkLeS+JeT+0o2XOsFZp7Uhr9yFIodRsnNo
   H/Uwmab28ocNrGNiEVaVH6eTTQeeZuOdoQzUbClElpVmkrNGY0M42K0PvOQ/e7+y
   AQ==
   -----END CERTIFICATE-----

2. Add the certificate to the client-certificate-authority-data field in the ${INSTALL_DIR}/auth/kubeconfig file.

Procedure

1. If you have physical access to the nodes, check their console output to determine if they have successfully booted. If the nodes are still retrieving their boot configuration, there might be problems with the bootstrap VM .
2. Ensure you have configured the sshKey: '<ssh_pub_key>' setting in the install-config.yaml file, where <ssh_pub_key> is the public key of the kni user on the provisioner node.

Procedure

1. Check the network connectivity to the provisioning network.
2. Ensure PXE is enabled on the NIC for the provisioning network and PXE is disabled for all other NICs.

3. Verify that the install-config.yaml configuration file includes the rootDeviceHints parameter and boot MAC address for the NIC connected to the provisioning network. For example:

   control plane node settings

   bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC

   Worker node settings

   bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC

Procedure

1. Check the Machine API Operator by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig \
      --namespace=openshift-machine-api get deployments

   If ${INSTALL_DIR} is not set in your environment, replace the value with the name of the installation directory.

   Example output

   NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
   cluster-autoscaler-operator   1/1     1            1           86m
   cluster-baremetal-operator    1/1     1            1           86m
   machine-api-controllers       1/1     1            1           85m
   machine-api-operator          1/1     1            1           86m

2. Check the machine controller logs by running the following command:

   $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig \
        --namespace=openshift-machine-api logs deployments/machine-api-controllers \
        --container=machine-controller

Procedure

1. Ensure the network configuration exists by running the following command:

   $ oc get network -o yaml cluster

   If it does not exist, the installation program did not create it. To find out why, run the following command:

   $ openshift-install create manifests

   Review the manifests to determine why the installation program did not create the network configuration.

2. Ensure the network is running by entering the following command:

   $ oc get po -n openshift-network-operator

Procedure

1. Add a DNS A/AAAA or CNAME record to internally identify the API load balancer. For example, when using dnsmasq, modify the dnsmasq.conf configuration file:

   $ sudo nano /etc/dnsmasq.conf

   address=/api-int.<cluster_name>.<base_domain>/<IP_address>
   address=/api-int.mycluster.example.com/192.168.1.10
   address=/api-int.mycluster.example.com/2001:0db8:85a3:0000:0000:8a2e:0370:7334

2. Add a DNS PTR record to internally identify the API load balancer. For example, when using dnsmasq, modify the dnsmasq.conf configuration file:

   $ sudo nano /etc/dnsmasq.conf

   ptr-record=<IP_address>.in-addr.arpa,api-int.<cluster_name>.<base_domain>
   ptr-record=10.1.168.192.in-addr.arpa,api-int.mycluster.example.com

3. Restart the DNS server. For example, when using dnsmasq, execute the following command:

   $ sudo systemctl restart dnsmasq

Procedure

1. Power off all bare-metal nodes before installing the OpenShift Container Platform cluster by using the following command:

   $ ipmitool -I lanplus -U <user> -P <password> -H <management_server_ip> power off

2. Remove all old bootstrap resources if any remain from an earlier deployment attempt by using the following script:

   for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
   do
     sudo virsh destroy $i;
     sudo virsh undefine $i;
     sudo virsh vol-delete $i --pool $i;
     sudo virsh vol-delete $i.ign --pool $i;
     sudo virsh pool-destroy $i;
     sudo virsh pool-undefine $i;
   done

3. Delete the artifacts that the earlier installation generated by using the following command:

   $ cd ; /bin/rm -rf auth/ bootstrap.ign master.ign worker.ign metadata.json \
   .openshift_install.log .openshift_install_state.json

4. Re-create the OpenShift Container Platform manifests by using the following command:

   $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests

Procedure

1. Check to ensure authentication is successful:

   $ /usr/local/bin/oc adm release mirror \
     -a pull-secret-update.json
     --from=$UPSTREAM_REPO \
     --to-release-image=$LOCAL_REG/$LOCAL_REPO:${VERSION} \
     --to=$LOCAL_REG/$LOCAL_REPO

   Note

   Example output of the variables used to mirror the install images:

   UPSTREAM_REPO=${RELEASE_IMAGE}
   LOCAL_REG=<registry_FQDN>:<registry_port>
   LOCAL_REPO='ocp4/openshift4'

   The values of RELEASE_IMAGE and VERSION were set during the Retrieving OpenShift Installer step of the Setting up the environment for an OpenShift installation section.

2. After mirroring the registry, confirm that you can access it in your disconnected environment:

   $ curl -k -u <user>:<password> https://registry.example.com:<registry_port>/v2/_catalog
   {"repositories":["<Repo_Name>"]}

Procedure

1. When the OpenShift Container Platform cluster nodes are installed appropriately, the following Ready state is seen within the STATUS column:

   $ oc get nodes

   NAME                   STATUS   ROLES           AGE  VERSION
   master-0.example.com   Ready    master,worker   4h   v1.30.3
   master-1.example.com   Ready    master,worker   4h   v1.30.3
   master-2.example.com   Ready    master,worker   4h   v1.30.3

2. Confirm the installation program deployed all pods successfully. The following command removes any pods that are still running or have completed as part of the output.

   $ oc get pods --all-namespaces | grep -iv running | grep -iv complete