Chapter 11. Contacting Red Hat support for service

1. Install the ceph-debuginfo package if it is not installed already.

   1. Enable the repository containing the ceph-debuginfo package:

      Red Hat Enterprise Linux 7:

      Copy to Clipboard Toggle word wrap

      subscription-manager repos --enable=rhel-7-server-rhceph-4-DAEMON-debug-rpms

      Replace DAEMON with osd or mon depending on the type of Ceph node.

      Red Hat Enterprise Linux 8:

      Copy to Clipboard Toggle word wrap

      subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-debug-rpms

   2. Install the ceph-debuginfo package:

      Copy to Clipboard Toggle word wrap

      [root@mon ~]# yum install ceph-debuginfo

2. Ensure that the gdb package is installed and if it is not, install it:

   Copy to Clipboard Toggle word wrap

   [root@mon ~]# yum install gdb

Procedure

1. Enable generating core dump files for Ceph.

   1. Set the proper ulimits for the core dump files by adding the following parameter to the /etc/systemd/system.conf file:

      Copy to Clipboard Toggle word wrap

      DefaultLimitCORE=infinity

   2. Comment out the PrivateTmp=true parameter in the Ceph daemon service file, by default located at /lib/systemd/system/CLUSTER_NAME-DAEMON@.service:

      Copy to Clipboard Toggle word wrap

      [root@mon ~]# PrivateTmp=true

   3. Set the suid_dumpable flag to 2 to allow the Ceph daemons to generate dump core files:

      Copy to Clipboard Toggle word wrap

      [root@mon ~]# sysctl fs.suid_dumpable=2

   4. Adjust the core dump files location:

      Copy to Clipboard Toggle word wrap

      [root@mon ~]# sysctl kernel.core_pattern=/tmp/core

   5. Modify /etc/systemd/coredump.conf file by adding the following lines under section [Coredump]:

      Copy to Clipboard Toggle word wrap

      ProcessSizeMax=8G
      ExternalSizeMax=8G
      JournalSizeMax=8G

   6. Reload the systemd service for the changes to take effect:

      Copy to Clipboard Toggle word wrap

      [root@mon ~]# systemctl daemon-reload

   7. Restart the Ceph daemon for the changes to take effect:

      Copy to Clipboard Toggle word wrap

      [root@mon ~]# systemctl restart ceph-DAEMON@ID

      Specify the daemon type (osd or mon) and its ID (numbers for OSDs, or short host names for Monitors) for example:

      Copy to Clipboard Toggle word wrap

      [root@mon ~]# systemctl restart ceph-osd@1

2. Reproduce the failure, for example try to start the daemon again.

3. Use the GNU Debugger (GDB) to generate a readable backtrace from an application core dump file:

   Copy to Clipboard Toggle word wrap

   gdb /usr/bin/ceph-DAEMON /tmp/core.PID

   Specify the daemon type and the PID of the failed process, for example:

   Copy to Clipboard Toggle word wrap

   $ gdb /usr/bin/ceph-osd /tmp/core.123456

   In the GDB command prompt disable paging and enable logging to a file by entering the commands set pag off and set log on:

   Copy to Clipboard Toggle word wrap

   (gdb) set pag off
   (gdb) set log on

   Apply the backtrace command to all threads of the process by entering thr a a bt full:

   Copy to Clipboard Toggle word wrap

   (gdb) thr a a bt full

   After the backtrace is generated turn off logging by entering set log off:

   Copy to Clipboard Toggle word wrap

   (gdb) set log off

4. Transfer the log file gdb.txt to the system you access the Red Hat Customer Portal from and attach it to a support ticket.

Procedure

1. If a Ceph process terminates unexpectedly due to the SIGILL, SIGTRAP, SIGABRT, or SIGSEGV error:

   1. Set the core pattern to the systemd-coredump service on the node where the container with the failed Ceph process is running, for example:

      Copy to Clipboard Toggle word wrap

      [root@mon]# echo "| /usr/lib/systemd/systemd-coredump %P %u %g %s %t %e" > /proc/sys/kernel/core_pattern

   2. Watch for the next container failure due to a Ceph process and search for a core dump file in the /var/lib/systemd/coredump/ directory, for example:

      Copy to Clipboard Toggle word wrap

      [root@mon]# ls -ltr /var/lib/systemd/coredump
      total 8232
      -rw-r-----. 1 root root 8427548 Jan 22 19:24 core.ceph-osd.167.5ede29340b6c4fe4845147f847514c12.15622.1584573794000000.xz

2. To manually capture a core dump file for the Ceph Monitors and Ceph Managers:

   1. Get the ceph-mon package details of the Ceph daemon from the container:

      Red Hat Enterprise Linux 7:

      Copy to Clipboard Toggle word wrap

      [root@mon]# docker exec -it NAME /bin/bash
      [root@mon]# rpm -qa | grep ceph

      Red Hat Enterprise Linux 8:

      Copy to Clipboard Toggle word wrap

      [root@mon]# podman exec -it NAME /bin/bash
      [root@mon]# rpm -qa | grep ceph

      Replace NAME with the name of the Ceph container.

   2. Make a backup copy and open for editing the ceph-mon@.service file:

      Copy to Clipboard Toggle word wrap

      [root@mon]# cp /etc/systemd/system/ceph-mon@.service /etc/systemd/system/ceph-mon@.service.orig

   3. In the ceph-mon@.service file, add these three options to the [Service] section, each on a separate line:

      Copy to Clipboard Toggle word wrap

      --pid=host \
      --ipc=host \
      --cap-add=SYS_PTRACE \

      Example

      Copy to Clipboard Toggle word wrap

      [Unit]
      Description=Ceph Monitor
      After=docker.service
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/docker rm ceph-mon-%i
      ExecStartPre=/bin/sh -c '"$(command -v mkdir)" -p /etc/ceph /var/lib/ceph/mon'
      ExecStart=/usr/bin/docker run --rm --name ceph-mon-%i \
        --memory=924m \
      --cpu-quota=100000 \
      -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
      -v /etc/localtime:/etc/localtime:ro \
      --net=host \
      --privileged=true \
      --ipc=host \ 

      1

      
      --pid=host \ 

      2

      
      --cap-add=SYS_PTRACE \ 

      3

      
      -e IP_VERSION=4 \
              -e MON_IP=10.74.131.17 \
            -e CLUSTER=ceph \
        -e FSID=9448efca-b1a1-45a3-bf7b-b55cba696a6e \
        -e CEPH_PUBLIC_NETWORK=10.74.131.0/24 \
        -e CEPH_DAEMON=MON \
         \
        registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest
      ExecStop=-/usr/bin/docker stop ceph-mon-%i
      ExecStopPost=-/bin/rm -f /var/run/ceph/ceph-mon.pd-cephcontainer-mon01.asok
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      
      [Install]
      WantedBy=multi-user.target

   4. Restart the Ceph Monitor daemon:

      Syntax

      Copy to Clipboard Toggle word wrap

      systemctl restart ceph-mon@MONITOR_ID

      Replace MONITOR_ID with the ID number of the Ceph Monitor.

      Example

      Copy to Clipboard Toggle word wrap

      [root@mon]# systemctl restart ceph-mon@1

   5. Install the gdb package inside the Ceph Monitor container:

      Red Hat Enterprise Linux 7:

      Copy to Clipboard Toggle word wrap

      [root@mon]# docker exec -it ceph-mon-MONITOR_ID /bin/bash
      sh $ yum install gdb

      Red Hat Enterprise Linux 8:

      Copy to Clipboard Toggle word wrap

      [root@mon]# podman exec -it ceph-mon-MONITOR_ID /bin/bash
      sh $ yum install gdb

      Replace MONITOR_ID with the ID number of the Ceph Monitor.

   6. Find the process ID:

      Syntax

      Copy to Clipboard Toggle word wrap

      ps -aef | grep PROCESS | grep -v run

      Replace PROCESS with the name of failed process, for example ceph-mon.

      Example

      Copy to Clipboard Toggle word wrap

      [root@mon]# ps -aef | grep ceph-mon | grep -v run
      ceph       15390   15266  0 18:54 ?        00:00:29 /usr/bin/ceph-mon --cluster ceph --setroot ceph --setgroup ceph -d -i 5
      ceph       18110   17985  1 19:40 ?        00:00:08 /usr/bin/ceph-mon --cluster ceph --setroot ceph --setgroup ceph -d -i 2

   7. Generate the core dump file:

      Syntax

      Copy to Clipboard Toggle word wrap

      gcore ID

      Replace ID with the ID of the failed process that you got from the previous step, for example 18110:

      Example

      Copy to Clipboard Toggle word wrap

      [root@mon]# gcore 18110
      warning: target file /proc/18110/cmdline contained unexpected null characters
      Saved corefile core.18110

   8. Verify that the core dump file has been generated correctly.

      Example

      Copy to Clipboard Toggle word wrap

      [root@mon]# ls -ltr
      total 709772
      -rw-r--r--. 1 root root 726799544 Mar 18 19:46 core.18110

   9. Copy the core dump file outside of the Ceph Monitor container:

      Red Hat Enterprise Linux 7:

      Copy to Clipboard Toggle word wrap

      [root@mon]# docker cp ceph-mon-MONITOR_ID:/tmp/mon.core.MONITOR_PID /tmp

      Red Hat Enterprise Linux 8:

      Copy to Clipboard Toggle word wrap

      [root@mon]# podman cp ceph-mon-MONITOR_ID:/tmp/mon.core.MONITOR_PID /tmp

      Replace MONITOR_ID with the ID number of the Ceph Monitor and replace MONITOR_PID with the process ID number.

   10. Restore the backup copy of the ceph-mon@.service file:

       Copy to Clipboard Toggle word wrap

       [root@mon]# cp /etc/systemd/system/ceph-mon@.service.orig /etc/systemd/system/ceph-mon@.service

   11. Restart the Ceph Monitor daemon:

       Syntax

       Copy to Clipboard Toggle word wrap

       systemctl restart ceph-mon@MONITOR_ID

       Replace MONITOR_ID with the ID number of the Ceph Monitor.

       Example

       Copy to Clipboard Toggle word wrap

       [root@mon]# systemctl restart ceph-mon@1

   12. Upload the core dump file for analysis by Red Hat support, see step 4.

3. To manually capture a core dump file for Ceph OSDs:

   1. Get the ceph-osd package details of the Ceph daemon from the container:

      Red Hat Enterprise Linux 7:

      Copy to Clipboard Toggle word wrap

      [root@osd]# docker exec -it NAME /bin/bash
      [root@osd]# rpm -qa | grep ceph

      Red Hat Enterprise Linux 8:

      Copy to Clipboard Toggle word wrap

      [root@osd]# podman exec -it NAME /bin/bash
      [root@osd]# rpm -qa | grep ceph

      Replace NAME with the name of the Ceph container.

   2. Install the Ceph package for the same version of the ceph-osd package on the node where the Ceph containers are running:

      Red Hat Enterprise Linux 7:

      Copy to Clipboard Toggle word wrap

      [root@osd]# yum install ceph-osd

      Red Hat Enterprise Linux 8:

      Copy to Clipboard Toggle word wrap

      [root@osd]# dnf install ceph-osd

      If needed, enable the appropriate repository first. See the Enabling the Red Hat Ceph Storage repositories section in the Installation Guide for details.

   3. Find the ID of the process that has failed:

      Copy to Clipboard Toggle word wrap

      ps -aef | grep PROCESS | grep -v run

      Replace PROCESS with the name of failed process, for example ceph-osd.

      Copy to Clipboard Toggle word wrap

      [root@osd]# ps -aef | grep ceph-osd | grep -v run
      ceph       15390   15266  0 18:54 ?        00:00:29 /usr/bin/ceph-osd --cluster ceph --setroot ceph --setgroup ceph -d -i 5
      ceph       18110   17985  1 19:40 ?        00:00:08 /usr/bin/ceph-osd --cluster ceph --setroot ceph --setgroup ceph -d -i 2

   4. Generate the core dump file:

      Copy to Clipboard Toggle word wrap

      gcore ID

      Replace ID with the ID of the failed process that you got from the previous step, for example 18110:

      Copy to Clipboard Toggle word wrap

      [root@osd]# gcore 18110
      warning: target file /proc/18110/cmdline contained unexpected null characters
      Saved corefile core.18110

   5. Verify that the core dump file has been generated correctly.

      Copy to Clipboard Toggle word wrap

      [root@osd]# ls -ltr
      total 709772
      -rw-r--r--. 1 root root 726799544 Mar 18 19:46 core.18110

   6. Upload the core dump file for analysis by Red Hat support, see the next step.
4. Upload the core dump file for analysis to a Red Hat support case. See Providing information to Red Hat Support engineers for details.