Chapter 2. Downloading log files and diagnostic information using must-gather
If Red Hat OpenShift Data Foundation is unable to automatically resolve a problem, use the must-gather tool to collect log files and diagnostic information so that you or Red Hat support can review the problem and determine a solution.
When Red Hat OpenShift Data Foundation is deployed in external mode, must-gather only collects logs from the OpenShift Data Foundation cluster and does not collect debug data and logs from the external Red Hat Ceph Storage cluster. To collect debug logs from the external Red Hat Ceph Storage cluster, see Red Hat Ceph Storage Troubleshooting guide and contact your Red Hat Ceph Storage Administrator.
Prerequisites
Optional: If OpenShift Data Foundation is deployed in a disconnected environment, ensure that you mirror the individual
must-gatherimage to the mirror registry available from the disconnected environment.$ oc image mirror registry.redhat.io/odf4/odf-must-gather-rhel9:v4.15 <local-registry>/odf4/odf-must-gather-rhel9:v4.15 [--registry-config=<path-to-the-registry-config>] [--insecure=true]
<local-registry>- Is the local image mirror registry available for a disconnected OpenShift Container Platform cluster.
<path-to-the-registry-config>-
Is the path to your registry credentials, by default it is
~/.docker/config.json. --insecure- Add this flag only if the mirror registry is insecure.
For more information, see the Red Hat Knowledgebase solutions:
Procedure
Run the
must-gathercommand from the client connected to the OpenShift Data Foundation cluster:$ oc adm must-gather --image=registry.redhat.io/odf4/odf-must-gather-rhel9:v4.15 --dest-dir=<directory-name><directory-name>Is the name of the directory where you want to write the data to.
ImportantFor a disconnected environment deployment, replace the image in
--imageparameter with the mirroredmust-gatherimage.$ oc adm must-gather --image=<local-registry>/odf4/odf-must-gather-rhel9:v4.15 --dest-dir=<directory-name><local-registry>- Is the local image mirror registry available for a disconnected OpenShift Container Platform cluster.
This collects the following information in the specified directory:
- All Red Hat OpenShift Data Foundation cluster related Custom Resources (CRs) with their namespaces.
- Pod logs of all the Red Hat OpenShift Data Foundation related pods.
- Output of some standard Ceph commands like Status, Cluster health, and others.
2.1. Variations of must-gather-commands
If one or more master nodes are not in the Ready state, use
--node-nameto provide a master node that is Ready so that themust-gatherpod can be safely scheduled.$ oc adm must-gather --image=registry.redhat.io/odf4/odf-must-gather-rhel9:v4.15 --dest-dir=_<directory-name>_ --node-name=_<node-name>_
If you want to gather information from a specific time:
To specify a relative time period for logs gathered, such as within 5 seconds or 2 days, add
/usr/bin/gather since=<duration>:$ oc adm must-gather --image=registry.redhat.io/odf4/odf-must-gather-rhel9:v4.15 --dest-dir=_<directory-name>_ /usr/bin/gather since=<duration>
To specify a specific time to gather logs after, add
/usr/bin/gather since-time=<rfc3339-timestamp>:$ oc adm must-gather --image=registry.redhat.io/odf4/odf-must-gather-rhel9:v4.15 --dest-dir=_<directory-name>_ /usr/bin/gather since-time=<rfc3339-timestamp>
Replace the example values in these commands as follows:
- <node-name>
-
If one or more master nodes are not in the Ready state, use this parameter to provide the name of a master node that is still in the Ready state. This avoids scheduling errors by ensuring that the
must-gatherpod is not scheduled on a master node that is not ready. - <directory-name>
-
The directory to store information collected by
must-gather. - <duration>
-
Specify the period of time to collect information from as a relative duration, for example,
5h(starting from 5 hours ago). - <rfc3339-timestamp>
-
Specify the period of time to collect information from as an RFC 3339 timestamp, for example,
2020-11-10T04:00:00+00:00(starting from 4 am UTC on 11 Nov 2020).
2.2. Running must-gather in modular mode
Red Hat OpenShift Data Foundation must-gather can take a long time to run in some environments. To avoid this, run must-gather in modular mode and collect only the resources you require using the following command:
$ oc adm must-gather --image=registry.redhat.io/odf4/odf-must-gather-rhel9:v4.15 -- /usr/bin/gather <-arg>
Replace < -arg> with one or more of the following arguments to specify the resources for which the must-gather logs is required.
-o,--odf- ODF logs (includes Ceph resources, namespaced resources, clusterscoped resources and Ceph logs)
-d,--dr- DR logs
-n,--noobaa- Noobaa logs
-c,--ceph- Ceph commands and pod logs
-cl,--ceph-logs- Ceph daemon, kernel, journal logs, and crash reports
-ns,--namespaced-
namespacedresources -cs,--clusterscoped-
clusterscopedresources -h,--help- Print help message
