Chapter 6. Gathering data about your cluster

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command:

   $ oc adm must-gather

   Note

   Because this command picks a random control plane node by default, the pod might be scheduled to a control plane node that is in the NotReady and SchedulingDisabled state.

   1. If this command fails, for example, if you cannot schedule a pod on your cluster, then use the oc adm inspect command to gather information for particular resources.

      Note

      Contact Red Hat Support for the recommended resources to gather.

3. Create a compressed file from the must-gather directory that was just created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 1

   1

   Make sure to replace must-gather-local.5421342344627712289/ with the actual directory name.

4. Attach the compressed file to your support case on the the Customer Support page of the Red Hat Customer Portal.

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Run the oc adm must-gather command with one or more --image or --image-stream arguments.

   Note
   • To collect the default must-gather data in addition to specific feature data, add the --image-stream=openshift/must-gather argument.

   For example, the following command gathers both the default cluster data and information specific to OpenShift Virtualization:

   $ oc adm must-gather \
     --image-stream=openshift/must-gather \ 1
     --image=registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.17.0 2

   1

   The default Red Hat OpenShift Service on AWS must-gather image

   2

   The must-gather image for OpenShift Virtualization

   You can use the must-gather tool with additional arguments to gather data that is specifically related to OpenShift Logging and the Red Hat OpenShift Logging Operator in your cluster. For OpenShift Logging, run the following command:

   $ oc adm must-gather --image=$(oc -n openshift-logging get deployment.apps/cluster-logging-operator \
     -o jsonpath='{.spec.template.spec.containers[?(@.name == "cluster-logging-operator")].image}')

   Example 6.1. Example must-gather output for OpenShift Logging

   ├── cluster-logging
   │  ├── clo
   │  │  ├── cluster-logging-operator-74dd5994f-6ttgt
   │  │  ├── clusterlogforwarder_cr
   │  │  ├── cr
   │  │  ├── csv
   │  │  ├── deployment
   │  │  └── logforwarding_cr
   │  ├── collector
   │  │  ├── fluentd-2tr64
   │  ├── eo
   │  │  ├── csv
   │  │  ├── deployment
   │  │  └── elasticsearch-operator-7dc7d97b9d-jb4r4
   │  ├── es
   │  │  ├── cluster-elasticsearch
   │  │  │  ├── aliases
   │  │  │  ├── health
   │  │  │  ├── indices
   │  │  │  ├── latest_documents.json
   │  │  │  ├── nodes
   │  │  │  ├── nodes_stats.json
   │  │  │  └── thread_pool
   │  │  ├── cr
   │  │  ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
   │  │  └── logs
   │  │     ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
   │  ├── install
   │  │  ├── co_logs
   │  │  ├── install_plan
   │  │  ├── olmo_logs
   │  │  └── subscription
   │  └── kibana
   │     ├── cr
   │     ├── kibana-9d69668d4-2rkvz
   ├── cluster-scoped-resources
   │  └── core
   │     ├── nodes
   │     │  ├── ip-10-0-146-180.eu-west-1.compute.internal.yaml
   │     └── persistentvolumes
   │        ├── pvc-0a8d65d9-54aa-4c44-9ecc-33d9381e41c1.yaml
   ├── event-filter.html
   ├── gather-debug.log
   └── namespaces
      ├── openshift-logging
      │  ├── apps
      │  │  ├── daemonsets.yaml
      │  │  ├── deployments.yaml
      │  │  ├── replicasets.yaml
      │  │  └── statefulsets.yaml
      │  ├── batch
      │  │  ├── cronjobs.yaml
      │  │  └── jobs.yaml
      │  ├── core
      │  │  ├── configmaps.yaml
      │  │  ├── endpoints.yaml
      │  │  ├── events
      │  │  │  ├── elasticsearch-im-app-1596020400-gm6nl.1626341a296c16a1.yaml
      │  │  │  ├── elasticsearch-im-audit-1596020400-9l9n4.1626341a2af81bbd.yaml
      │  │  │  ├── elasticsearch-im-infra-1596020400-v98tk.1626341a2d821069.yaml
      │  │  │  ├── elasticsearch-im-app-1596020400-cc5vc.1626341a3019b238.yaml
      │  │  │  ├── elasticsearch-im-audit-1596020400-s8d5s.1626341a31f7b315.yaml
      │  │  │  ├── elasticsearch-im-infra-1596020400-7mgv8.1626341a35ea59ed.yaml
      │  │  ├── events.yaml
      │  │  ├── persistentvolumeclaims.yaml
      │  │  ├── pods.yaml
      │  │  ├── replicationcontrollers.yaml
      │  │  ├── secrets.yaml
      │  │  └── services.yaml
      │  ├── openshift-logging.yaml
      │  ├── pods
      │  │  ├── cluster-logging-operator-74dd5994f-6ttgt
      │  │  │  ├── cluster-logging-operator
      │  │  │  │  └── cluster-logging-operator
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  └── cluster-logging-operator-74dd5994f-6ttgt.yaml
      │  │  ├── cluster-logging-operator-registry-6df49d7d4-mxxff
      │  │  │  ├── cluster-logging-operator-registry
      │  │  │  │  └── cluster-logging-operator-registry
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── cluster-logging-operator-registry-6df49d7d4-mxxff.yaml
      │  │  │  └── mutate-csv-and-generate-sqlite-db
      │  │  │     └── mutate-csv-and-generate-sqlite-db
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── elasticsearch-cdm-lp8l38m0-1-794d6dd989-4jxms
      │  │  ├── elasticsearch-im-app-1596030300-bpgcx
      │  │  │  ├── elasticsearch-im-app-1596030300-bpgcx.yaml
      │  │  │  └── indexmanagement
      │  │  │     └── indexmanagement
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── fluentd-2tr64
      │  │  │  ├── fluentd
      │  │  │  │  └── fluentd
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── fluentd-2tr64.yaml
      │  │  │  └── fluentd-init
      │  │  │     └── fluentd-init
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  │  ├── kibana-9d69668d4-2rkvz
      │  │  │  ├── kibana
      │  │  │  │  └── kibana
      │  │  │  │     └── logs
      │  │  │  │        ├── current.log
      │  │  │  │        ├── previous.insecure.log
      │  │  │  │        └── previous.log
      │  │  │  ├── kibana-9d69668d4-2rkvz.yaml
      │  │  │  └── kibana-proxy
      │  │  │     └── kibana-proxy
      │  │  │        └── logs
      │  │  │           ├── current.log
      │  │  │           ├── previous.insecure.log
      │  │  │           └── previous.log
      │  └── route.openshift.io
      │     └── routes.yaml
      └── openshift-operators-redhat
         ├── ...

3. Run the oc adm must-gather command with one or more --image or --image-stream arguments. For example, the following command gathers both the default cluster data and information specific to KubeVirt:

   $ oc adm must-gather \
    --image-stream=openshift/must-gather \ 1
    --image=quay.io/kubevirt/must-gather 2

   1

   The default Red Hat OpenShift Service on AWS must-gather image

   2

   The must-gather image for KubeVirt

4. Create a compressed file from the must-gather directory that was just created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather.tar.gz must-gather.local.5421342344627712289/ 1

   1

   Make sure to replace must-gather-local.5421342344627712289/ with the actual directory name.

5. Attach the compressed file to your support case on the the Customer Support page of the Red Hat Customer Portal.

Procedure

1. Run the oc adm must-gather command with -- gather_network_logs:

   $ oc adm must-gather -- gather_network_logs

   Note

   By default, the must-gather tool collects the OVN nbdb and sbdb databases from all of the nodes in the cluster. Adding the -- gather_network_logs option to include additional logs that contain OVN-Kubernetes transactions for OVN nbdb database.

2. Create a compressed file from the must-gather directory that was just created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather.tar.gz must-gather.local.472290403699006248 1

   1

   Replace must-gather-local.472290403699006248 with the actual directory name.

3. Attach the compressed file to your support case on the the Customer Support page of the Red Hat Customer Portal.

Procedure

1. Run a packet capture from the host network on some nodes by running the following command:

   $ oc adm must-gather \
       --dest-dir /tmp/captures \  <.>
       --source-dir '/tmp/tcpdump/' \  <.>
       --image registry.redhat.io/openshift4/network-tools-rhel8:latest \  <.>
       --node-selector 'node-role.kubernetes.io/worker' \  <.>
       --host-network=true \  <.>
       --timeout 30s \  <.>
       -- \
       tcpdump -i any \  <.>
       -w /tmp/tcpdump/%Y-%m-%dT%H:%M:%S.pcap -W 1 -G 300

   <.> The --dest-dir argument specifies that oc adm must-gather stores the packet captures in directories that are relative to /tmp/captures on the client machine. You can specify any writable directory. <.> When tcpdump is run in the debug pod that oc adm must-gather starts, the --source-dir argument specifies that the packet captures are temporarily stored in the /tmp/tcpdump directory on the pod. <.> The --image argument specifies a container image that includes the tcpdump command. <.> The --node-selector argument and example value specifies to perform the packet captures on the worker nodes. As an alternative, you can specify the --node-name argument instead to run the packet capture on a single node. If you omit both the --node-selector and the --node-name argument, the packet captures are performed on all nodes. <.> The --host-network=true argument is required so that the packet captures are performed on the network interfaces of the node. <.> The --timeout argument and value specify to run the debug pod for 30 seconds. If you do not specify the --timeout argument and a duration, the debug pod runs for 10 minutes. <.> The -i any argument for the tcpdump command specifies to capture packets on all network interfaces. As an alternative, you can specify a network interface name.

2. Perform the action, such as accessing a web application, that triggers the network communication issue while the network trace captures packets.

3. Review the packet capture files that oc adm must-gather transferred from the pods to your client machine:

   tmp/captures
   ├── event-filter.html
   ├── ip-10-0-192-217-ec2-internal  1
   │   └── registry-redhat-io-openshift4-network-tools-rhel8-sha256-bca...
   │       └── 2022-01-13T19:31:31.pcap
   ├── ip-10-0-201-178-ec2-internal  2
   │   └── registry-redhat-io-openshift4-network-tools-rhel8-sha256-bca...
   │       └── 2022-01-13T19:31:30.pcap
   ├── ip-...
   └── timestamp

   1 2

   The packet captures are stored in directories that identify the hostname, container, and file name. If you did not specify the --node-selector argument, then the directory level for the hostname is not present.

Procedure

1. Obtain a list of cluster nodes:

   $ oc get nodes

2. Enter into a debug session on the target node. This step instantiates a debug pod called <node_name>-debug:

   $ oc debug node/my-cluster-node

3. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

   # chroot /host

4. From within the chroot environment console, obtain the node’s interface names:

   # ip ad

5. Start a toolbox container, which includes the required binaries and plugins to run sosreport:

   # toolbox

   Note

   If an existing toolbox pod is already running, the toolbox command outputs 'toolbox-' already exists. Trying to start…​. To avoid tcpdump issues, remove the running toolbox container with podman rm toolbox- and spawn a new toolbox container.

6. Initiate a tcpdump session on the cluster node and redirect output to a capture file. This example uses ens5 as the interface name:

   $ tcpdump -nn -s 0 -i ens5 -w /host/var/tmp/my-cluster-node_$(date +%d_%m_%Y-%H_%M_%S-%Z).pcap  1

   1

   The tcpdump capture file’s path is outside of the chroot environment because the toolbox container mounts the host’s root directory at /host.

7. If a tcpdump capture is required for a specific container on the node, follow these steps.

   1. Determine the target container ID. The chroot host command precedes the crictl command in this step because the toolbox container mounts the host’s root directory at /host:

      # chroot /host crictl ps

   2. Determine the container’s process ID. In this example, the container ID is a7fe32346b120:

      # chroot /host crictl inspect --output yaml a7fe32346b120 | grep 'pid' | awk '{print $2}'

   3. Initiate a tcpdump session on the container and redirect output to a capture file. This example uses 49628 as the container’s process ID and ens5 as the interface name. The nsenter command enters the namespace of a target process and runs a command in its namespace. because the target process in this example is a container’s process ID, the tcpdump command is run in the container’s namespace from the host:

      # nsenter -n -t 49628 -- tcpdump -nn -i ens5 -w /host/var/tmp/my-cluster-node-my-container_$(date +%d_%m_%Y-%H_%M_%S-%Z).pcap  1

      1

      The tcpdump capture file’s path is outside of the chroot environment because the toolbox container mounts the host’s root directory at /host.

8. Provide the tcpdump capture file to Red Hat Support for analysis, using one of the following methods.

   • Upload the file to an existing Red Hat support case directly from an Red Hat OpenShift Service on AWS cluster.

     1. From within the toolbox container, run redhat-support-tool to attach the file directly to an existing Red Hat Support case. This example uses support case ID 01234567:

        # redhat-support-tool addattachment -c 01234567 /host/var/tmp/my-tcpdump-capture-file.pcap 1

        1

        The toolbox container mounts the host’s root directory at /host. Reference the absolute path from the toolbox container’s root directory, including /host/, when specifying files to upload through the redhat-support-tool command.

   • Upload the file to an existing Red Hat support case.

     1. Concatenate the sosreport archive by running the oc debug node/<node_name> command and redirect the output to a file. This command assumes you have exited the previous oc debug session:

        $ oc debug node/my-cluster-node -- bash -c 'cat /host/var/tmp/my-tcpdump-capture-file.pcap' > /tmp/my-tcpdump-capture-file.pcap 1

        1

        The debug container mounts the host’s root directory at /host. Reference the absolute path from the debug container’s root directory, including /host, when specifying target files for concatenation.

     2. Navigate to an existing support case within the Customer Support page of the Red Hat Customer Portal.
     3. Select Attach files and follow the prompts to upload the file.

Procedure

1. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

   # chroot /host

2. Start the toolbox container:

   # toolbox

3. Install the additional package, such as wget:

   # dnf install -y <package_name>

Procedure

1. Set /host as the root directory within the debug shell. The debug pod mounts the host’s root file system in /host within the pod. By changing the root directory to /host, you can run binaries contained in the host’s executable paths:

   # chroot /host

2. Create a .toolboxrc file in the home directory for the root user ID:

   # vi ~/.toolboxrc

   REGISTRY=quay.io                1
   IMAGE=fedora/fedora:33-x86_64   2
   TOOLBOX_NAME=toolbox-fedora-33  3

   1

   Optional: Specify an alternative container registry.

   2

   Specify an alternative image to start.

   3

   Optional: Specify an alternative name for the toolbox container.

3. Start a toolbox container with the alternative image:

   # toolbox

   Note

   If an existing toolbox pod is already running, the toolbox command outputs 'toolbox-' already exists. Trying to start…​. Remove the running toolbox container with podman rm toolbox- and spawn a new toolbox container, to avoid issues with sosreport plugins.