Chapter 3. Commonly required logs for troubleshooting

• Generating logs for a specific pod:

  Copy to Clipboard Toggle word wrap

   $ oc logs <pod-name> -n <namespace>

• Generating logs for Ceph or OpenShift Container Storage cluster:

  Copy to Clipboard Toggle word wrap

  $ oc logs rook-ceph-operator-<ID> -n openshift-storage

  Important

  Currently, the rook-ceph-operator logs do not provide any information about the failure and this acts as a limitation in troubleshooting issues, see Enabling and disabling debug logs for rook-ceph-operator.

• Generating logs for plugin pods like cephfs or rbd to detect any problem in the PVC mount of the app-pod:

  Copy to Clipboard Toggle word wrap

  $ oc logs csi-cephfsplugin-<ID> -n openshift-storage -c csi-cephfsplugin

  Copy to Clipboard Toggle word wrap

  $ oc logs csi-rbdplugin-<ID> -n openshift-storage -c csi-rbdplugin

  • To generate logs for all the containers in the CSI pod:

    Copy to Clipboard Toggle word wrap

    $ oc logs csi-cephfsplugin-<ID> -n openshift-storage --all-containers

    Copy to Clipboard Toggle word wrap

    $ oc logs csi-rbdplugin-<ID> -n openshift-storage --all-containers

• Generating logs for cephfs or rbd provisioner pods to detect problems if PVC is not in BOUND state:

  Copy to Clipboard Toggle word wrap

  $ oc logs csi-cephfsplugin-provisioner-<ID> -n openshift-storage -c csi-cephfsplugin

  Copy to Clipboard Toggle word wrap

  $ oc logs csi-rbdplugin-provisioner-<ID> -n openshift-storage -c csi-rbdplugin

  • To generate logs for all the containers in the CSI pod:

    Copy to Clipboard Toggle word wrap

    $ oc logs csi-cephfsplugin-provisioner-<ID> -n openshift-storage --all-containers

    Copy to Clipboard Toggle word wrap

    $ oc logs csi-rbdplugin-provisioner-<ID> -n openshift-storage --all-containers

• Generating OpenShift Container Storage logs using cluster-info command:

  Copy to Clipboard Toggle word wrap

  $ oc cluster-info dump -n openshift-storage --output-directory=<directory-name>

• Check the OpenShift Container Storage operator logs and events.

  • To check the operator logs :

    Copy to Clipboard Toggle word wrap

    # oc logs <ocs-operator> -n openshift-storage

    <ocs-operator>

    Copy to Clipboard Toggle word wrap

    # oc get pods -n openshift-storage | grep -i "ocs-operator" | awk '{print $1}'

  • To check the operator events :

    Copy to Clipboard Toggle word wrap

    # oc get events --sort-by=metadata.creationTimestamp -n openshift-storage

• Get the OpenShift Container Storage operator version and channel.

  Copy to Clipboard Toggle word wrap

  # oc get csv -n openshift-storage

  Example output :

  Copy to Clipboard Toggle word wrap

  NAME                     DISPLAY VERSION              REPLACES
  PHASE
  ocs-operator.v4.7.2      OpenShift Container Storage  4.7.2
  Succeeded

  Copy to Clipboard Toggle word wrap

  # oc get subs -n openshift-storage

  Example output :

  Copy to Clipboard Toggle word wrap

  NAME          PACKAGE        SOURCE
  CHANNEL
  ocs-operator  ocs-operator   redhat-operators
  stable-4.8

• Confirm that the installplan is created.

  Copy to Clipboard Toggle word wrap

  # oc get installplan -n openshift-storage

• Verify the image of the components post updating OpenShift Container Storage.

  • Check the node on which the pod of the component you want to verify the image is running.

    Copy to Clipboard Toggle word wrap

    # oc get pods -o wide | grep <component-name>

    For Example :

    Copy to Clipboard Toggle word wrap

    # oc get pods -o wide | grep rook-ceph-operator

    Example output:

    Copy to Clipboard Toggle word wrap

    rook-ceph-operator-566cc677fd-bjqnb 1/1 Running 20 4h6m 10.128.2.5 rook-ceph-operator-566cc677fd-bjqnb 1/1 Running 20 4h6m 10.128.2.5 dell-r440-12.gsslab.pnq2.redhat.com <none> <none>
    
    <none> <none>

    dell-r440-12.gsslab.pnq2.redhat.com is the node-name.

  • Check the image ID.

    Copy to Clipboard Toggle word wrap

    # oc debug node/<node name>

    <node-name>

    Is the name of the node on which the pod of the component you want to verify the image is running.

    Copy to Clipboard Toggle word wrap

    # chroot /host

    Copy to Clipboard Toggle word wrap

    # crictl images | grep <component>

    For Example :

    Copy to Clipboard Toggle word wrap

    # crictl images | grep rook-ceph

    Example output:

    Copy to Clipboard Toggle word wrap

    IMAGE                                                     TAG
          IMAGEID          SIZE
    registry.redhat.io/ocs4/rook-ceph-rhel8-operator@sha256   <none>
          5600a36370df4    1.55GB

    Take a note of the IMAGEID and map it to the Digest ID on the Rook Ceph Operator page.

Additional resources

• Using must-gather