Chapter 5. Gathering data about your cluster

Procedure

1. Obtain a list of cluster nodes:

   $ oc get nodes

2. Enter into a debug session on the target node. This step instantiates a debug pod called

   <node_name>-debug

   :

   $ oc debug node/my-cluster-node

   To enter into a debug session on the target node that is tainted with the

   NoExecute

   effect, add a toleration to a dummy namespace, and start the debug pod in the dummy namespace:

   $ oc new-project dummy

   $ oc patch namespace dummy --type=merge -p '{"metadata": {"annotations": { "scheduler.alpha.kubernetes.io/defaultTolerations": "[{\"operator\": \"Exists\"}]"}}}'

   $ oc debug node/my-cluster-node

3. Set

   /host

   as the root directory within the debug shell. The debug pod mounts the host’s root file system in

   /host

   within the pod. By changing the root directory to

   /host

   , you can run binaries contained in the host’s executable paths:

   # chroot /host

   Note

   OpenShift Container Platform 4.11 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node,

   oc

   operations will be impacted. In such situations, it is possible to access nodes using

   ssh core@<node>.<cluster_name>.<base_domain>

   instead.

4. Start a

   toolbox

   container, which includes the required binaries and plugins to run

   sosreport

   :

   # toolbox

   Note

   If an existing

   toolbox

   pod is already running, the

   toolbox

   command outputs

   'toolbox-' already exists. Trying to start…​

   . Remove the running toolbox container with

   podman rm toolbox-

   and spawn a new toolbox container, to avoid issues with

   sosreport

   plugins.

5. Collect a

   sosreport

   archive.

   1. Run the

      sosreport

      command and enable the

      crio.all

      and

      crio.logs

      CRI-O container engine

      sosreport

      plugins:

      # sosreport -k crio.all=on -k crio.logs=on 

      1

      1

      -k enables you to define sosreport plugin parameters outside of the defaults.

   2. Press Enter when prompted, to continue.
   3. Provide the Red Hat Support case ID.

      sosreport

      adds the ID to the archive’s file name.

   4. The

      sosreport

      output provides the archive’s location and checksum. The following sample output references support case ID

      01234567

      :

      Your sosreport has been generated and saved in:
        /host/var/tmp/sosreport-my-cluster-node-01234567-2020-05-28-eyjknxt.tar.xz 

      1

      
      
      The checksum is: 382ffc167510fd71b4f12a4f40b97a4e

      1

      The sosreport archive’s file path is outside of the chroot environment because the toolbox container mounts the host’s root directory at /host.

6. Provide the

   sosreport

   archive to Red Hat Support for analysis, using one of the following methods.

   • Upload the file to an existing Red Hat support case directly from an OpenShift Container Platform cluster.

     1. From within the toolbox container, run

        redhat-support-tool

        to attach the archive directly to an existing Red Hat support case. This example uses support case ID

        01234567

        :

        # redhat-support-tool addattachment -c 01234567 /host/var/tmp/my-sosreport.tar.xz 

        1

        1

        The toolbox container mounts the host’s root directory at /host. Reference the absolute path from the toolbox container’s root directory, including /host/, when specifying files to upload through the redhat-support-tool command.

   • Upload the file to an existing Red Hat support case.

     1. Concatenate the

        sosreport

        archive by running the

        oc debug node/<node_name>

        command and redirect the output to a file. This command assumes you have exited the previous

        oc debug

        session:

        $ oc debug node/my-cluster-node -- bash -c 'cat /host/var/tmp/sosreport-my-cluster-node-01234567-2020-05-28-eyjknxt.tar.xz' > /tmp/sosreport-my-cluster-node-01234567-2020-05-28-eyjknxt.tar.xz 

        1

        1

        The debug container mounts the host’s root directory at /host. Reference the absolute path from the debug container’s root directory, including /host, when specifying target files for concatenation.

        Note

        OpenShift Container Platform 4.11 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Transferring a

        sosreport

        archive from a cluster node by using

        scp

        is not recommended. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node,

        oc

        operations will be impacted. In such situations, it is possible to copy a

        sosreport

        archive from a node by running

        scp core@<node>.<cluster_name>.<base_domain>:<file_path> <local_path>

        .

     2. Navigate to an existing support case within the Customer Support page of the Red Hat Customer Portal.
     3. Select Attach files and follow the prompts to upload the file.

Procedure

1. Query

   bootkube.service

   journald

   unit logs from a bootstrap node during OpenShift Container Platform installation. Replace

   <bootstrap_fqdn>

   with the bootstrap node’s fully qualified domain name:

   $ ssh core@<bootstrap_fqdn> journalctl -b -f -u bootkube.service

   Note

   The

   bootkube.service

   log on the bootstrap node outputs etcd

   connection refused

   errors, indicating that the bootstrap server is unable to connect to etcd on control plane nodes. After etcd has started on each control plane node and the nodes have joined the cluster, the errors should stop.

2. Collect logs from the bootstrap node containers using

   podman

   on the bootstrap node. Replace

   <bootstrap_fqdn>

   with the bootstrap node’s fully qualified domain name:

   $ ssh core@<bootstrap_fqdn> 'for pod in $(sudo podman ps -a -q); do sudo podman logs $pod; done'

Procedure

1. Query

   kubelet

   journald

   unit logs from OpenShift Container Platform cluster nodes. The following example queries control plane nodes only:

   $ oc adm node-logs --role=master -u kubelet  

   1

   1

   Replace kubelet as appropriate to query other unit logs.

2. Collect logs from specific subdirectories under

   /var/log/

   on cluster nodes.

   1. Retrieve a list of logs contained within a

      /var/log/

      subdirectory. The following example lists files in

      /var/log/openshift-apiserver/

      on all control plane nodes:

      $ oc adm node-logs --role=master --path=openshift-apiserver

   2. Inspect a specific log within a

      /var/log/

      subdirectory. The following example outputs

      /var/log/openshift-apiserver/audit.log

      contents from all control plane nodes:

      $ oc adm node-logs --role=master --path=openshift-apiserver/audit.log

   3. If the API is not functional, review the logs on each node using SSH instead. The following example tails

      /var/log/openshift-apiserver/audit.log

      :

      $ ssh core@<master-node>.<cluster_name>.<base_domain> sudo tail -f /var/log/openshift-apiserver/audit.log

      Note

      OpenShift Container Platform 4.11 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. Before attempting to collect diagnostic data over SSH, review whether the data collected by running

      oc adm must gather

      and other

      oc

      commands is sufficient instead. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node,

      oc

      operations will be impacted. In such situations, it is possible to access nodes using

      ssh core@<node>.<cluster_name>.<base_domain>

      .

Procedure

1. Run a packet capture from the host network on some nodes by running the following command:

   $ oc adm must-gather \
       --dest-dir /tmp/captures \  <.>
       --source-dir '/tmp/tcpdump/' \  <.>
       --image registry.redhat.io/openshift4/network-tools-rhel8:latest \  <.>
       --node-selector 'node-role.kubernetes.io/worker' \  <.>
       --host-network=true \  <.>
       --timeout 30s \  <.>
       -- \
       tcpdump -i any \  <.>
       -w /tmp/tcpdump/%Y-%m-%dT%H:%M:%S.pcap -W 1 -G 300

   <.> The

   --dest-dir

   argument specifies that

   oc adm must-gather

   stores the packet captures in directories that are relative to

   /tmp/captures

   on the client machine. You can specify any writable directory. <.> When

   tcpdump

   is run in the debug pod that

   oc adm must-gather

   starts, the

   --source-dir

   argument specifies that the packet captures are temporarily stored in the

   /tmp/tcpdump

   directory on the pod. <.> The

   --image

   argument specifies a container image that includes the

   tcpdump

   command. <.> The

   --node-selector

   argument and example value specifies to perform the packet captures on the worker nodes. As an alternative, you can specify the

   --node-name

   argument instead to run the packet capture on a single node. If you omit both the

   --node-selector

   and the

   --node-name

   argument, the packet captures are performed on all nodes. <.> The

   --host-network=true

   argument is required so that the packet captures are performed on the network interfaces of the node. <.> The

   --timeout

   argument and value specify to run the debug pod for 30 seconds. If you do not specify the

   --timeout

   argument and a duration, the debug pod runs for 10 minutes. <.> The

   -i any

   argument for the

   tcpdump

   command specifies to capture packets on all network interfaces. As an alternative, you can specify a network interface name.

2. Perform the action, such as accessing a web application, that triggers the network communication issue while the network trace captures packets.

3. Review the packet capture files that

   oc adm must-gather

   transferred from the pods to your client machine:

   tmp/captures
   ├── event-filter.html
   ├── ip-10-0-192-217-ec2-internal  

   1

   
   │   └── registry-redhat-io-openshift4-network-tools-rhel8-sha256-bca...
   │       └── 2022-01-13T19:31:31.pcap
   ├── ip-10-0-201-178-ec2-internal  

   2

   
   │   └── registry-redhat-io-openshift4-network-tools-rhel8-sha256-bca...
   │       └── 2022-01-13T19:31:30.pcap
   ├── ip-...
   └── timestamp

   1 2

   The packet captures are stored in directories that identify the hostname, container, and file name. If you did not specify the --node-selector argument, then the directory level for the hostname is not present.

Procedure

1. Obtain a list of cluster nodes:

   $ oc get nodes

2. Enter into a debug session on the target node. This step instantiates a debug pod called

   <node_name>-debug

   :

   $ oc debug node/my-cluster-node

3. Set

   /host

   as the root directory within the debug shell. The debug pod mounts the host’s root file system in

   /host

   within the pod. By changing the root directory to

   /host

   , you can run binaries contained in the host’s executable paths:

   # chroot /host

   Note

   OpenShift Container Platform 4.11 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Accessing cluster nodes by using SSH is not recommended. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node,

   oc

   operations will be impacted. In such situations, it is possible to access nodes using

   ssh core@<node>.<cluster_name>.<base_domain>

   instead.

4. From within the

   chroot

   environment console, obtain the node’s interface names:

   # ip ad

5. Start a

   toolbox

   container, which includes the required binaries and plugins to run

   sosreport

   :

   # toolbox

   Note

   If an existing

   toolbox

   pod is already running, the

   toolbox

   command outputs

   'toolbox-' already exists. Trying to start…​

   . To avoid

   tcpdump

   issues, remove the running toolbox container with

   podman rm toolbox-

   and spawn a new toolbox container.

6. Initiate a

   tcpdump

   session on the cluster node and redirect output to a capture file. This example uses

   ens5

   as the interface name:

   $ tcpdump -nn -s 0 -i ens5 -w /host/var/tmp/my-cluster-node_$(date +%d_%m_%Y-%H_%M_%S-%Z).pcap  

   1

   1

   The tcpdump capture file’s path is outside of the chroot environment because the toolbox container mounts the host’s root directory at /host.

7. If a

   tcpdump

   capture is required for a specific container on the node, follow these steps.

   1. Determine the target container ID. The

      chroot host

      command precedes the

      crictl

      command in this step because the toolbox container mounts the host’s root directory at

      /host

      :

      # chroot /host crictl ps

   2. Determine the container’s process ID. In this example, the container ID is

      a7fe32346b120

      :

      # chroot /host crictl inspect --output yaml a7fe32346b120 | grep 'pid' | awk '{print $2}'

   3. Initiate a

      tcpdump

      session on the container and redirect output to a capture file. This example uses

      49628

      as the container’s process ID and

      ens5

      as the interface name. The

      nsenter

      command enters the namespace of a target process and runs a command in its namespace. because the target process in this example is a container’s process ID, the

      tcpdump

      command is run in the container’s namespace from the host:

      # nsenter -n -t 49628 -- tcpdump -nn -i ens5 -w /host/var/tmp/my-cluster-node-my-container_$(date +%d_%m_%Y-%H_%M_%S-%Z).pcap  

      1

      1

      The tcpdump capture file’s path is outside of the chroot environment because the toolbox container mounts the host’s root directory at /host.

8. Provide the

   tcpdump

   capture file to Red Hat Support for analysis, using one of the following methods.

   • Upload the file to an existing Red Hat support case directly from an OpenShift Container Platform cluster.

     1. From within the toolbox container, run

        redhat-support-tool

        to attach the file directly to an existing Red Hat Support case. This example uses support case ID

        01234567

        :

        # redhat-support-tool addattachment -c 01234567 /host/var/tmp/my-tcpdump-capture-file.pcap 

        1

        1

        The toolbox container mounts the host’s root directory at /host. Reference the absolute path from the toolbox container’s root directory, including /host/, when specifying files to upload through the redhat-support-tool command.

   • Upload the file to an existing Red Hat support case.

     1. Concatenate the

        sosreport

        archive by running the

        oc debug node/<node_name>

        command and redirect the output to a file. This command assumes you have exited the previous

        oc debug

        session:

        $ oc debug node/my-cluster-node -- bash -c 'cat /host/var/tmp/my-tcpdump-capture-file.pcap' > /tmp/my-tcpdump-capture-file.pcap 

        1

        1

        The debug container mounts the host’s root directory at /host. Reference the absolute path from the debug container’s root directory, including /host, when specifying target files for concatenation.

        Note

        OpenShift Container Platform 4.11 cluster nodes running Red Hat Enterprise Linux CoreOS (RHCOS) are immutable and rely on Operators to apply cluster changes. Transferring a

        tcpdump

        capture file from a cluster node by using

        scp

        is not recommended. However, if the OpenShift Container Platform API is not available, or the kubelet is not properly functioning on the target node,

        oc

        operations will be impacted. In such situations, it is possible to copy a

        tcpdump

        capture file from a node by running

        scp core@<node>.<cluster_name>.<base_domain>:<file_path> <local_path>

        .

     2. Navigate to an existing support case within the Customer Support page of the Red Hat Customer Portal.
     3. Select Attach files and follow the prompts to upload the file.

Procedure

1. Set

   /host

   as the root directory within the debug shell. The debug pod mounts the host’s root file system in

   /host

   within the pod. By changing the root directory to

   /host

   , you can run binaries contained in the host’s executable paths:

   # chroot /host

2. Start the toolbox container:

   # toolbox

3. Install the additional package, such as

   wget

   :

   # dnf install -y <package_name>

Procedure

1. Set

   /host

   as the root directory within the debug shell. The debug pod mounts the host’s root file system in

   /host

   within the pod. By changing the root directory to

   /host

   , you can run binaries contained in the host’s executable paths:

   # chroot /host

2. Create a

   .toolboxrc

   file in the home directory for the root user ID:

   # vi ~/.toolboxrc

   REGISTRY=quay.io                

   1

   
   IMAGE=fedora/fedora:33-x86_64   

   2

   
   TOOLBOX_NAME=toolbox-fedora-33  

   3

   1

   Optional: Specify an alternative container registry.

   2

   Specify an alternative image to start.

   3

   Optional: Specify an alternative name for the toolbox container.

3. Start a toolbox container with the alternative image:

   # toolbox

   Note

   If an existing

   toolbox

   pod is already running, the

   toolbox

   command outputs

   'toolbox-' already exists. Trying to start…​

   . Remove the running toolbox container with

   podman rm toolbox-

   and spawn a new toolbox container, to avoid issues with

   sosreport

   plugins.