5.2. Verifying node health

Procedure

1. List the name, status, and role for all nodes in the cluster:

   $ oc get nodes

2. Summarize CPU and memory usage for each node within the cluster:

   $ oc adm top nodes

3. Summarize CPU and memory usage for a specific node:

   $ oc adm top node -l my-node

Procedure

1. The kubelet is managed using a systemd service on each node. Review the kubelet’s status by querying the kubelet systemd service within a debug pod.

   1. Start a debug pod for a node:

      $ oc debug node/my-node

   2. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

      # chroot /host

      注意

      OpenShift Container Platform cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes using SSH is not recommended and nodes will be tainted as accessed. However, if the OpenShift Container Platform API is not available, or kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain> instead.

   3. Check whether the kubelet systemd service is active on the node:

      # systemctl is-active kubelet

   4. Output a more detailed kubelet.service status summary:

      # systemctl status kubelet

Procedure

1. Query kubelet journald unit logs from OpenShift Container Platform cluster nodes. The following example queries master nodes only:

   $ oc adm node-logs --role=master -u kubelet  1

   1

   Replace kubelet as appropriate to query other unit logs.

2. Collect logs from specific subdirectories under /var/log/ on cluster nodes.

   1. Retrieve a list of logs contained within a /var/log/ subdirectory. The following example lists files in /var/log/openshift-apiserver/ on all master nodes:

      $ oc adm node-logs --role=master --path=openshift-apiserver

   2. Inspect a specific log within a /var/log/ subdirectory. The following example outputs /var/log/openshift-apiserver/audit.log contents from all master nodes:

      $ oc adm node-logs --role=master --path=openshift-apiserver/audit.log

   3. If the API is not functional, review the logs on each node using SSH instead. The following example tails /var/log/openshift-apiserver/audit.log:

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo tail -f /var/log/openshift-apiserver/audit.log

      注意

      OpenShift Container Platform 4.5 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes using SSH is not recommended and nodes will be tainted as accessed. Before attempting to collect diagnostic data over SSH, review whether the data collected by running oc adm must gather and other oc commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node, oc operations will be impacted. In such situations, it is possible to access nodes using ssh core@<node>.<cluster_name>.<base_domain>.