Chapter 7. Troubleshooting the Ceph iSCSI gateway (Limited Availability)

Procedure

1. On the VWware ESXi node, open the kernel log:

   [root@esx:~]# more /var/log/vmkernel.log

   Copy to Clipboard Toggle word wrap

2. Gather information from the following error messages in the VMware ESXi kernel log:

   Example

   2022-05-30T11:07:07.570Z cpu32:66506)iscsi_vmk:
   iscsivmk_ConnRxNotifyFailure: Sess [ISID: 00023d000005 TARGET:
   iqn.2017-12.com.redhat.iscsi-gw:ceph-igw TPGT: 3 TSIH: 0]

   Copy to Clipboard Toggle word wrap

   From this message, make a note of the ISID number, the TARGET name, and the Target Portal Group Tag (TPGT) number. For this example, we have the following:

   ISID: 00023d000005
   TARGET: iqn.2017-12.com.redhat.iscsi-gw:ceph-igw
   TPGT: 3

   Copy to Clipboard Toggle word wrap

   Example

   2022-05-30T11:07:07.570Z cpu32:66506)iscsi_vmk:
   iscsivmk_ConnRxNotifyFailure: vmhba64:CH:4 T:0 CN:0: Connection rx
   notifying failure: Failed to Receive. State=Bound

   Copy to Clipboard Toggle word wrap

   From this message, make a note of the adapter channel (CH) number. For this example, we have the following:

   vmhba64:CH:4 T:0

   Copy to Clipboard Toggle word wrap

3. To find the remote address of the Ceph iSCSI gateway node:

   [root@esx:~]# esxcli iscsi session connection list

   Copy to Clipboard Toggle word wrap

   Example

   ...
   vmhba64,iqn.2017-12.com.redhat.iscsi-gw:ceph-igw,00023d000003,0
      Adapter: vmhba64
      Target: iqn.2017-12.com.redhat.iscsi-gw:ceph-igw 

   1

   
      ISID: 00023d000003 

   2

   
      CID: 0
      DataDigest: NONE
      HeaderDigest: NONE
      IFMarker: false
      IFMarkerInterval: 0
      MaxRecvDataSegmentLength: 131072
      MaxTransmitDataSegmentLength: 262144
      OFMarker: false
      OFMarkerInterval: 0
      ConnectionAddress: 10.2.132.2
      RemoteAddress: 10.2.132.2 

   3

   
      LocalAddress: 10.2.128.77
      SessionCreateTime: 03/28/18 21:45:19
      ConnectionCreateTime: 03/28/18 21:45:19
      ConnectionStartTime: 03/28/18 21:45:19
      State: xpt_wait
   ...

   Copy to Clipboard Toggle word wrap

   From the command output, match the ISID value, and the TARGET name value gathered previously, then make a note of the RemoteAddress value. From this example, we have the following:

   Target: iqn.2017-12.com.redhat.iscsi-gw:ceph-igw
   ISID: 00023d000003
   RemoteAddress: 10.2.132.2

   Copy to Clipboard Toggle word wrap

   Now, you can collect more information from the Ceph iSCSI gateway node to further troubleshoot the issue.

   1. On the Ceph iSCSI gateway node mentioned by the RemoteAddress value, run an sosreport to gather system information:

      [root@igw ~]# sosreport

      Copy to Clipboard Toggle word wrap

4. To find a disk that went into a dead state:

   [root@esx:~]# esxcli storage nmp device list

   Copy to Clipboard Toggle word wrap

   Example

   ...
   iqn.1998-01.com.vmware:d04-nmgjd-pa-zyc-sv039-rh2288h-xnh-732d78fd-00023d000004,iqn.2017-12.com.redhat.iscsi-gw:ceph-igw,t,3-naa.60014054a5d46697f85498e9a257567c
      Runtime Name: vmhba64:C4:T0:L4 

   1

   
      Device: naa.60014054a5d46697f85498e9a257567c 

   2

   
      Device Display Name: LIO-ORG iSCSI Disk
   (naa.60014054a5d46697f85498e9a257567c)
      Group State: dead 

   3

   
      Array Priority: 0
      Storage Array Type Path Config:
   {TPG_id=3,TPG_state=ANO,RTP_id=3,RTP_health=DOWN} 

   4

   
      Path Selection Policy Path Config: {non-current path; rank: 0}
   ...

   Copy to Clipboard Toggle word wrap

   From the command output, match the CH number, and the TPGT number gathered previously, then make a note of the Device value. For this example, we have the following:

   vmhba64:C4:T0
   Device: naa.60014054a5d46697f85498e9a257567c
   TPG_id=3

   Copy to Clipboard Toggle word wrap

   With the device name, you can gather some additional information on each iSCSI disk in a dead state.

   1. Gather more information on the iSCSI disk:

      Syntax

      esxcli storage nmp path list -d ISCSI_DISK_DEVICE > /tmp/esxcli_storage_nmp_path_list.txt
      esxcli storage core device list -d ISCSI_DISK_DEVICE > /tmp/esxcli_storage_core_device_list.txt

      Copy to Clipboard Toggle word wrap

      Example

      [root@esx:~]# esxcli storage nmp path list -d naa.60014054a5d46697f85498e9a257567c > /tmp/esxcli_storage_nmp_path_list.txt
      [root@esx:~]# esxcli storage core device list -d naa.60014054a5d46697f85498e9a257567c > /tmp/esxcli_storage_core_device_list.txt

      Copy to Clipboard Toggle word wrap

5. Gather additional information on the VMware ESXi environment:

   [root@esx:~]# esxcli storage vmfs extent list > /tmp/esxcli_storage_vmfs_extent_list.txt
   [root@esx:~]# esxcli storage filesystem list > /tmp/esxcli_storage_filesystem_list.txt
   [root@esx:~]# esxcli iscsi session list > /tmp/esxcli_iscsi_session_list.txt
   [root@esx:~]# esxcli iscsi session connection list > /tmp/esxcli_iscsi_session_connection_list.txt

   Copy to Clipboard Toggle word wrap

6. Check for potential iSCSI login issues:

   • Was the iSCSI login data not sent?
   • Did the iSCSI login timeout or fail to find a portal group?

Procedure

1. Enable additional logging:

   [root@igw ~]# echo "iscsi_target_mod +p" > /sys/kernel/debug/dynamic_debug/control
   [root@igw ~]# echo "target_core_mod +p" > /sys/kernel/debug/dynamic_debug/control

   Copy to Clipboard Toggle word wrap
2. Wait a couple of minutes for the extra debugging information to populate the system log.

3. Disable the additional logging:

   [root@igw ~]# echo "iscsi_target_mod -p" > /sys/kernel/debug/dynamic_debug/control
   [root@igw ~]# echo "target_core_mod -p" > /sys/kernel/debug/dynamic_debug/control

   Copy to Clipboard Toggle word wrap

4. Run an sosreport to gather system information:

   [root@igw ~]# sosreport

   Copy to Clipboard Toggle word wrap

5. Capture network traffic for the Ceph iSCSI gateway and the VMware ESXi nodes simultaneously:

   Syntax

   tcpdump -s0 -i NETWORK_INTERFACE -w OUTPUT_FILE_PATH

   Copy to Clipboard Toggle word wrap

   Example

   [root@igw ~]# tcpdump -s 0 -i eth0 -w /tmp/igw-eth0-tcpdump.pcap

   Copy to Clipboard Toggle word wrap

   Note

   Look for traffic on port 3260.

   1. Network packet capture files can be large, so compress the tcpdump output from the iSCSI target and initiators before uploading any files to Red Hat Global Support Services:

      Syntax

      gzip OUTPUT_FILE_PATH

      Copy to Clipboard Toggle word wrap

      Example

      [root@igw ~]# gzip /tmp/igw-eth0-tcpdump.pcap

      Copy to Clipboard Toggle word wrap

6. Gather additional information on the VMware ESXi environment:

   [root@esx:~]# esxcli iscsi session list > /tmp/esxcli_iscsi_session_list.txt
   [root@esx:~]# esxcli iscsi session connection list > /tmp/esxcli_iscsi_session_connection_list.txt

   Copy to Clipboard Toggle word wrap

   1. List and collect more information on each iSCSI disk:

      Syntax

      esxcli storage nmp path list -d ISCSI_DISK_DEVICE > /tmp/esxcli_storage_nmp_path_list.txt

      Copy to Clipboard Toggle word wrap

      Example

      [root@esx:~]# esxcli storage nmp device list
      [root@esx:~]# esxcli storage nmp path list -d naa.60014054a5d46697f85498e9a257567c > /tmp/esxcli_storage_nmp_path_list.txt
      [root@esx:~]# esxcli storage core device list -d naa.60014054a5d46697f85498e9a257567c > /tmp/esxcli_storage_core_device_list.txt

      Copy to Clipboard Toggle word wrap

Procedure

1. Enable the dumping of waiting tasks and write them to a file:

   [root@igw ~]# dmesg -c ; echo w > /proc/sysrq-trigger ; dmesg -c > /tmp/waiting-tasks.txt

   Copy to Clipboard Toggle word wrap

2. Review the list of waiting tasks for the following messages:

   • iscsit_tpg_disable_portal_group
   • core_tmr_abort_task
   • transport_generic_free_cmd

   If any of these messages appear in the waiting task list, then this is an indication that something went wrong with the tcmu-runner service. Maybe the tcmu-runner service was not restarted properly, or maybe the tcmu-runner service has crashed.

3. Verify if the tcmu-runner service is running:

   [root@igw ~]# systemctl status tcmu-runner

   Copy to Clipboard Toggle word wrap

   1. If the tcmu-runner service is not running, then stop the rbd-target-gw service before restarting the tcmu-runner service:

      [root@igw ~]# systemctl stop rbd-target-gw
      [root@igw ~]# systemctl stop tcmu-runner
      [root@igw ~]# systemctl start tcmu-runner
      [root@igw ~]# systemctl start rbd-target-gw

      Copy to Clipboard Toggle word wrap
      Important

      Stopping the Ceph iSCSI gateway first prevents IOs from getting stuck while the tcmu-runner service is down.

   2. If the tcmu-runner service is running, the this might be a new bug. Open a new Red Hat support case.

To Troubleshoot This Problem

1. Check to see if there are waiting tasks that might be holding things up.
2. Check the health of the Red Hat Ceph Storage cluster.
3. Collect system information from each device in the path from the Ceph iSCSI gateway node to the iSCSI initiator node.

To Troubleshoot This Problem

1. Check for any network disruptions in the environment.
2. Check the health of the Red Hat Ceph Storage cluster.
3. Collect system information from each device in the path from the Ceph iSCSI gateway node to the iSCSI initiator node.