5.2. Verifying node health

Procedure

1. List the name, status, and role for all nodes in the cluster:

   $ oc get nodes

   Copy to Clipboard Toggle word wrap

2. Summarize CPU and memory usage for each node within the cluster:

   $ oc adm top nodes

   Copy to Clipboard Toggle word wrap

3. Summarize CPU and memory usage for a specific node:

   $ oc adm top node -l my-node

   Copy to Clipboard Toggle word wrap

Procedure

1. The kubelet is managed using a systemd service on each node. Review the kubelet’s status by querying the kubelet systemd service within a debug pod.

   1. Start a debug pod for a node:

      $ oc debug node/my-node

      Copy to Clipboard Toggle word wrap

   2. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

      # chroot /host

      Copy to Clipboard Toggle word wrap
      注意

      OpenShift Container Platform cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes using SSH is not recommended and nodes will be tainted as accessed. However, if the OpenShift Container Platform API is not available, or kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

   3. Check whether the kubelet systemd service is active on the node:

      # systemctl is-active kubelet

      Copy to Clipboard Toggle word wrap

   4. Output a more detailed kubelet.service status summary:

      # systemctl status kubelet

      Copy to Clipboard Toggle word wrap

Procedure

1. Query kubelet journald unit logs from OpenShift Container Platform cluster nodes. The following example queries master nodes only:

   $ oc adm node-logs --role=master -u kubelet  

   1

   Copy to Clipboard Toggle word wrap

   1

   Replace kubelet as appropriate to query other unit logs.

2. Collect logs from specific subdirectories under /var/log/ on cluster nodes.

   1. Retrieve a list of logs contained within a /var/log/ subdirectory. The following example lists files in /var/log/openshift-apiserver/ on all master nodes:

      $ oc adm node-logs --role=master --path=openshift-apiserver

      Copy to Clipboard Toggle word wrap

   2. Inspect a specific log within a /var/log/ subdirectory. The following example outputs /var/log/openshift-apiserver/audit.log contents from all master nodes:

      $ oc adm node-logs --role=master --path=openshift-apiserver/audit.log

      Copy to Clipboard Toggle word wrap

   3. If the API is not functional, review the logs on each node using SSH instead. The following example tails /var/log/openshift-apiserver/audit.log:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo tail -f /var/log/openshift-apiserver/audit.log

      Copy to Clipboard Toggle word wrap
      注意

      OpenShift Container Platform 4.5 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes using SSH is not recommended and nodes will be tainted as accessed. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.