Chapter 10. Contacting Red Hat support for service

Procedure

1. If a Ceph process terminates unexpectedly due to the SIGILL, SIGTRAP, SIGABRT, or SIGSEGV error:

   1. Set the core pattern to the systemd-coredump service on the node where the container with the failed Ceph process is running:

      Example

      [root@mon]# echo "| /usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" > /proc/sys/kernel/core_pattern

      Copy to Clipboard Toggle word wrap

   2. Watch for the next container failure due to a Ceph process and search for the core dump file in the /var/lib/systemd/coredump/ directory:

      Example

      [root@mon]# ls -ltr /var/lib/systemd/coredump
      total 8232
      -rw-r-----. 1 root root 8427548 Jan 22 19:24 core.ceph-osd.167.5ede29340b6c4fe4845147f847514c12.15622.1584573794000000.xz

      Copy to Clipboard Toggle word wrap

2. To manually capture a core dump file for the Ceph Monitors and Ceph OSDs:

   1. Get the MONITOR_ID or the OSD_ID and enter the container:

      Syntax

      podman ps
      podman exec -it MONITOR_ID_OR_OSD_ID bash

      Copy to Clipboard Toggle word wrap

      Example

      [root@host01 ~]# podman ps
      [root@host01 ~]# podman exec -it ceph-1ca9f6a8-d036-11ec-8263-fa163ee967ad-osd-2 bash

      Copy to Clipboard Toggle word wrap

   2. Install the procps-ng and gdb packages inside the container:

      Example

      [root@host01 ~]# dnf install procps-ng gdb

      Copy to Clipboard Toggle word wrap

   3. Find the process ID:

      Syntax

      ps -aef | grep PROCESS | grep -v run

      Copy to Clipboard Toggle word wrap

      Replace PROCESS with the name of the running process, for example ceph-mon or ceph-osd.

      Example

      [root@host01 ~]# ps -aef | grep ceph-mon | grep -v run
      ceph       15390   15266  0 18:54 ?        00:00:29 /usr/bin/ceph-mon --cluster ceph --setroot ceph --setgroup ceph -d -i 5
      ceph       18110   17985  1 19:40 ?        00:00:08 /usr/bin/ceph-mon --cluster ceph --setroot ceph --setgroup ceph -d -i 2

      Copy to Clipboard Toggle word wrap

   4. Generate the core dump file:

      Syntax

      gcore ID

      Copy to Clipboard Toggle word wrap

      Replace ID with the ID of the process that you got from the previous step, for example 18110:

      Example

      [root@host01 ~]# gcore 18110
      warning: target file /proc/18110/cmdline contained unexpected null characters
      Saved corefile core.18110

      Copy to Clipboard Toggle word wrap

   5. Verify that the core dump file has been generated correctly.

      Example

      [root@host01 ~]# ls -ltr
      total 709772
      -rw-r--r--. 1 root root 726799544 Mar 18 19:46 core.18110

      Copy to Clipboard Toggle word wrap

   6. Copy the core dump file outside of the Ceph Monitor container:

      Syntax

      podman cp ceph-mon-MONITOR_ID:/tmp/mon.core.MONITOR_PID /tmp

      Copy to Clipboard Toggle word wrap

      Replace MONITOR_ID with the ID number of the Ceph Monitor and replace MONITOR_PID with the process ID number.

3. To manually capture a core dump file for other Ceph daemons:

   1. Log in to the cephadm shell:

      Example

      [root@host03 ~]# cephadm shell

      Copy to Clipboard Toggle word wrap

   2. Enable ptrace for the daemons:

      Example

      [ceph: root@host01 /]# ceph config set mgr mgr/cephadm/allow_ptrace true

      Copy to Clipboard Toggle word wrap

   3. Redeploy the daemon service:

      Syntax

      ceph orch redeploy SERVICE_ID

      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph orch redeploy mgr
      [ceph: root@host01 /]# ceph orch redeploy rgw.rgw.1

      Copy to Clipboard Toggle word wrap

   4. Exit the cephadm shell and log in to the host where the daemons are deployed:

      Example

      [ceph: root@host01 /]# exit
      [root@host01 ~]# ssh root@10.0.0.11

      Copy to Clipboard Toggle word wrap

   5. Get the DAEMON_ID and enter the container:

      Example

      [root@host04 ~]# podman ps
      [root@host04 ~]# podman exec -it ceph-1ca9f6a8-d036-11ec-8263-fa163ee967ad-rgw-rgw-1-host04 bash

      Copy to Clipboard Toggle word wrap

   6. Install the procps-ng and gdb packages:

      Example

      [root@host04 /]# dnf install procps-ng gdb

      Copy to Clipboard Toggle word wrap

   7. Get the PID of process:

      Example

      [root@host04 /]# ps aux | grep rados
      ceph           6  0.3  2.8 5334140 109052 ?      Sl   May10   5:25 /usr/bin/radosgw -n client.rgw.rgw.1.host04 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug

      Copy to Clipboard Toggle word wrap

   8. Gather core dump:

      Syntax

      gcore PID

      Copy to Clipboard Toggle word wrap

      Example

      [root@host04 /]# gcore 6

      Copy to Clipboard Toggle word wrap

   9. Verify that the core dump file has been generated correctly.

      Example

      [root@host04 /]# ls -ltr
      total 108798
      -rw-r--r--. 1 root root 726799544 Mar 18 19:46 core.6

      Copy to Clipboard Toggle word wrap

   10. Copy the core dump file outside the container:

       Syntax

       podman cp ceph-mon-DAEMON_ID:/tmp/mon.core.PID /tmp

       Copy to Clipboard Toggle word wrap

       Replace DAEMON_ID with the ID number of the Ceph daemon and replace PID with the process ID number.

4. To allow systemd-coredump to successfully store the core dump for a crashed ceph daemon:

   1. Set DefaultLimitCORE to infinity in /etc/systemd/system.conf to allow core dump collection for a crashed process:

      Syntax

      # cat /etc/systemd/system.conf
      
      DefaultLimitCORE=infinity

      Copy to Clipboard Toggle word wrap

   2. Restart systemd or the node to apply the updated systemd settings:

      Syntax

      # sudo systemctl daemon-reexec

      Copy to Clipboard Toggle word wrap

   3. Verify the core dump files associated with previous daemon crashes:

      Syntax

      # ls -ltr /var/lib/systemd/coredump/

      Copy to Clipboard Toggle word wrap

5. Upload the core dump file for analysis to a Red Hat support case. See Providing information to Red Hat Support engineers for details.