Chapter 5. Troubleshooting the installation

Procedure

1. Log in to the bootstrap VM:

   $ ssh core@172.22.0.2

   Copy to Clipboard Toggle word wrap

2. To check the container logs, execute the following:

   [core@localhost ~]$ sudo podman logs -f <container_name>

   Copy to Clipboard Toggle word wrap

   Replace <container_name> with one of ironic or ironic-inspector. If you encounter an issue where the control plane nodes are not booting up from PXE, check the ironic pod. The ironic pod contains information about the attempt to boot the cluster nodes, because it attempts to log in to the node over IPMI.

Procedure

1. Log in to the bootstrap VM:

   $ ssh core@172.22.0.2

   Copy to Clipboard Toggle word wrap

2. Check the status of the coreos-downloader container within the bootstrap VM by running the following command:

   [core@localhost ~]$ sudo podman logs -f coreos-downloader

   Copy to Clipboard Toggle word wrap

   If the bootstrap VM cannot access the URL to the images, use the curl command to verify that the VM can access the images.

3. To inspect the bootkube logs that indicate if all the containers launched during the deployment phase, execute the following:

   [core@localhost ~]$ journalctl -xe

   Copy to Clipboard Toggle word wrap

   [core@localhost ~]$ journalctl -b -f -u bootkube.service

   Copy to Clipboard Toggle word wrap

4. Verify all the pods, including dnsmasq, mariadb, httpd, and ironic, are running:

   [core@localhost ~]$ sudo podman ps

   Copy to Clipboard Toggle word wrap

5. If there are issues with the pods, check the logs of the containers with issues. To check the logs of the ironic service, run the following command:

   [core@localhost ~]$ sudo podman logs ironic

   Copy to Clipboard Toggle word wrap

Procedure

1. Inspect the pods in the openshift-network-operator namespace:

   $ oc get all -n openshift-network-operator

   Copy to Clipboard Toggle word wrap

   NAME                                    READY STATUS            RESTARTS   AGE
   pod/network-operator-69dfd7b577-bg89v   0/1   ContainerCreating 0          149m

   Copy to Clipboard Toggle word wrap

2. On the provisioner node, determine that the network configuration exists:

   $ kubectl get network.config.openshift.io cluster -oyaml

   Copy to Clipboard Toggle word wrap

   apiVersion: config.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     serviceNetwork:
     - 172.30.0.0/16
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     networkType: OVNKubernetes

   Copy to Clipboard Toggle word wrap

   If it does not exist, the installer did not create it. To determine why the installer did not create it, execute the following:

   $ openshift-install create manifests

   Copy to Clipboard Toggle word wrap

3. Check that the network-operator is running:

   $ kubectl -n openshift-network-operator get pods

   Copy to Clipboard Toggle word wrap

4. Retrieve the logs:

   $ kubectl -n openshift-network-operator logs -l "name=network-operator"

   Copy to Clipboard Toggle word wrap

   On high availability clusters with three or more control plane (master) nodes, the Operator will perform leader election and all other Operators will sleep. For additional details, see Troubleshooting.

1. Ensure the reserved IPv6 addresses reside outside the DHCP range.

2. In the IP address reservation on the DHCP server, ensure the reservation specifies the correct DHCP Unique Identifier (DUID). For example:

   # This is a dnsmasq dhcp reservation, 'id:00:03:00:01' is the client id and '18:db:f2:8c:d5:9f' is the MAC Address for the NIC
   id:00:03:00:01:18:db:f2:8c:d5:9f,openshift-master-1,[2620:52:0:1302::6]

   Copy to Clipboard Toggle word wrap
3. Ensure that route announcements are working.
4. Ensure that the DHCP server is listening on the required interfaces serving the IP address ranges.

Procedure

1. Retrieve the hostname:

   [core@master-X ~]$ hostname

   Copy to Clipboard Toggle word wrap

   If the hostname is localhost, proceed with the following steps.

   Note

   Where X is the control plane node number.

2. Force the cluster node to renew the DHCP lease:

   [core@master-X ~]$ sudo nmcli con up "<bare_metal_nic>"

   Copy to Clipboard Toggle word wrap

   Replace <bare_metal_nic> with the wired connection corresponding to the baremetal network.

3. Check hostname again:

   [core@master-X ~]$ hostname

   Copy to Clipboard Toggle word wrap

4. If the hostname is still localhost.localdomain, restart NetworkManager:

   [core@master-X ~]$ sudo systemctl restart NetworkManager

   Copy to Clipboard Toggle word wrap
5. If the hostname is still localhost.localdomain, wait a few minutes and check again. If the hostname remains localhost.localdomain, repeat the previous steps.

6. Restart the nodeip-configuration service:

   [core@master-X ~]$ sudo systemctl restart nodeip-configuration.service

   Copy to Clipboard Toggle word wrap

   This service will reconfigure the kubelet service with the correct hostname references.

7. Reload the unit files definition since the kubelet changed in the previous step:

   [core@master-X ~]$ sudo systemctl daemon-reload

   Copy to Clipboard Toggle word wrap

8. Restart the kubelet service:

   [core@master-X ~]$ sudo systemctl restart kubelet.service

   Copy to Clipboard Toggle word wrap

9. Ensure kubelet booted with the correct hostname:

   [core@master-X ~]$ sudo journalctl -fu kubelet.service

   Copy to Clipboard Toggle word wrap

Addressing a csr

1. Get CSRs on the cluster:

   $ oc get csr

   Copy to Clipboard Toggle word wrap

2. Verify if a pending csr contains Subject Name: localhost.localdomain:

   $ oc get csr <pending_csr> -o jsonpath='{.spec.request}' | base64 --decode | openssl req -noout -text

   Copy to Clipboard Toggle word wrap

3. Remove any csr that contains Subject Name: localhost.localdomain:

   $ oc delete csr <wrong_csr>

   Copy to Clipboard Toggle word wrap

1. Get the route:

   $ oc get route oauth-openshift

   Copy to Clipboard Toggle word wrap

2. Check the service endpoint:

   $ oc get svc oauth-openshift

   Copy to Clipboard Toggle word wrap

   NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
   oauth-openshift   ClusterIP   172.30.19.162   <none>        443/TCP   59m

   Copy to Clipboard Toggle word wrap

3. Attempt to reach the service from a control plane (master) node:

   [core@master0 ~]$ curl -k https://172.30.19.162

   Copy to Clipboard Toggle word wrap

   {
     "kind": "Status",
     "apiVersion": "v1",
     "metadata": {
     },
     "status": "Failure",
     "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
     "reason": "Forbidden",
     "details": {
     },
     "code": 403

   Copy to Clipboard Toggle word wrap

4. Identify the authentication-operator errors from the provisioner node:

   $ oc logs deployment/authentication-operator -n openshift-authentication-operator

   Copy to Clipboard Toggle word wrap

   Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"225c5bd5-b368-439b-9155-5fd3c0459d98", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting"

   Copy to Clipboard Toggle word wrap

Solution

1. Ensure that the cluster name for every deployment is unique, ensuring no conflict.
2. Turn off all the rogue nodes which are not part of the cluster deployment that are using the same cluster name. Otherwise, the authentication pod of the OpenShift Container Platform cluster might never start successfully.

Procedure

1. Connect to the node where the Ignition configuration failed:

   Failed Units: 1
     machine-config-daemon-firstboot.service

   Copy to Clipboard Toggle word wrap

2. Restart the machine-config-daemon-firstboot service:

   [core@worker-X ~]$ sudo systemctl restart machine-config-daemon-firstboot.service

   Copy to Clipboard Toggle word wrap

Procedure

1. Check for differences in the AGE of the cluster nodes. For example:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

   NAME                         STATUS   ROLES    AGE   VERSION
   master-0.cloud.example.com   Ready    master   145m   v1.27.3
   master-1.cloud.example.com   Ready    master   135m   v1.27.3
   master-2.cloud.example.com   Ready    master   145m   v1.27.3
   worker-2.cloud.example.com   Ready    worker   100m   v1.27.3

   Copy to Clipboard Toggle word wrap

2. Check for inconsistent timing delays due to clock drift. For example:

   $ oc get bmh -n openshift-machine-api

   Copy to Clipboard Toggle word wrap

   master-1   error registering master-1  ipmi://<out_of_band_ip>

   Copy to Clipboard Toggle word wrap

   $ sudo timedatectl

   Copy to Clipboard Toggle word wrap

                  Local time: Tue 2020-03-10 18:20:02 UTC
              Universal time: Tue 2020-03-10 18:20:02 UTC
                    RTC time: Tue 2020-03-10 18:36:53
                   Time zone: UTC (UTC, +0000)
   System clock synchronized: no
                 NTP service: active
             RTC in local TZ: no

   Copy to Clipboard Toggle word wrap

Addressing clock drift in existing clusters

1. Create a Butane config file including the contents of the chrony.conf file to be delivered to the nodes. In the following example, create 99-master-chrony.bu to add the file to the control plane nodes. You can modify the file for worker nodes or repeat this procedure for the worker role.

   Note

   See "Creating machine configs with Butane" for information about Butane.

   variant: openshift
   version: 4.14.0
   metadata:
     name: 99-master-chrony
     labels:
       machineconfiguration.openshift.io/role: master
   storage:
     files:
     - path: /etc/chrony.conf
       mode: 0644
       overwrite: true
       contents:
         inline: |
           server <NTP_server> iburst 

   1

   
           stratumweight 0
           driftfile /var/lib/chrony/drift
           rtcsync
           makestep 10 3
           bindcmdaddress 127.0.0.1
           bindcmdaddress ::1
           keyfile /etc/chrony.keys
           commandkey 1
           generatecommandkey
           noclientlog
           logchange 0.5
           logdir /var/log/chrony

   Copy to Clipboard Toggle word wrap

   1

   Replace <NTP_server> with the IP address of the NTP server.

2. Use Butane to generate a MachineConfig object file, 99-master-chrony.yaml, containing the configuration to be delivered to the nodes:

   $ butane 99-master-chrony.bu -o 99-master-chrony.yaml

   Copy to Clipboard Toggle word wrap

3. Apply the MachineConfig object file:

   $ oc apply -f 99-master-chrony.yaml

   Copy to Clipboard Toggle word wrap

4. Ensure the System clock synchronized value is yes:

   $ sudo timedatectl

   Copy to Clipboard Toggle word wrap

                  Local time: Tue 2020-03-10 19:10:02 UTC
              Universal time: Tue 2020-03-10 19:10:02 UTC
                    RTC time: Tue 2020-03-10 19:36:53
                   Time zone: UTC (UTC, +0000)
   System clock synchronized: yes
                 NTP service: active
             RTC in local TZ: no

   Copy to Clipboard Toggle word wrap

   To setup clock synchronization prior to deployment, generate the manifest files and add this file to the openshift directory. For example:

   $ cp chrony-masters.yaml ~/clusterconfigs/openshift/99_masters-chrony-configuration.yaml

   Copy to Clipboard Toggle word wrap

   Then, continue to create the cluster.