이 콘텐츠는 선택한 언어로 제공되지 않습니다.

Chapter 9. Contacting Red Hat Support Service


If the information in this guide did not help you to solve the problem, this chapter explains how you contact the Red Hat Support Service.

9.1. Providing Information to Red Hat Support Engineers

If you are unable to fix problems related to Red Hat Ceph Storage by yourself, contact the Red Hat Support Service and provide sufficient amount of information that helps the support engineers to faster troubleshoot the problem you encounter.

Procedure: Providing Information to Red Hat Support Engineers

  1. Open a support ticket on the Red Hat Customer Portal.
  2. Ideally, attach an sosreport to the ticket. See the What is a sosreport and how to create one in Red Hat Enterprise Linux 4.6 and later? solution for details.
  3. If the Ceph daemons failed with a segmentation fault, consider generating a human-readable core dump file. See Section 9.2, “Generating readable core dump files” for details.

9.2. Generating readable core dump files

When a Ceph daemon terminates unexpectedly with a segmentation fault, gather the information about its failure and provide it to the Red Hat Support Engineers.

Such information speeds up the initial investigation. Also, the Support Engineers can compare the information from the core dump files with {storage-product} cluster known issues.

9.2.1. Prerequisites

  1. Install the ceph-debuginfo package if it is not installed already.

    1. Enable the repository containing the ceph-debuginfo package:

      subscription-manager repos --enable=rhel-7-server-rhceph-3-DAEMON-debug-rpms
      Copy to Clipboard Toggle word wrap

      Replace DAEMON with osd or mon depending on the type of the node.

    2. Install the ceph-debuginfo package:

      [root@mon ~]# yum install ceph-debuginfo
      Copy to Clipboard Toggle word wrap
  2. Ensure that the gdb package is installed and if it is not, install it:

    [root@mon ~]# yum install gdb
    Copy to Clipboard Toggle word wrap

Continue with the procedure based on the type of your deployment:

9.2.2. Generating readable core dump files on bare-metal deployments

Follow this procedure to generate a core dump file if you use Red Hat Ceph Storage on bare-metal.

Procedure

  1. Enable generating core dump files for Ceph.

    1. Set the proper ulimits for the core dump files by adding the following parameter to the /etc/systemd/system.conf file:

      DefaultLimitCORE=infinity
      Copy to Clipboard Toggle word wrap
    2. Comment out the PrivateTmp=true parameter in the Ceph daemon service file, by default located at /lib/systemd/system/CLUSTER_NAME-DAEMON@.service:

      [root@mon ~]# PrivateTmp=true
      Copy to Clipboard Toggle word wrap
    3. Set the suid_dumpable flag to 2 to allow the Ceph daemons to generate dump core files:

      [root@mon ~]# sysctl fs.suid_dumpable=2
      Copy to Clipboard Toggle word wrap
    4. Adjust the core dump files location:

      [root@mon ~]# sysctl kernel.core_pattern=/tmp/core
      Copy to Clipboard Toggle word wrap
    5. Reload the systemd service for the changes to take effect:

      [root@mon ~]# systemctl daemon-reload
      Copy to Clipboard Toggle word wrap
    6. Restart the Ceph daemon for the changes to take effect:

      [root@mon ~]# systemctl restart ceph-DAEMON@ID
      Copy to Clipboard Toggle word wrap

      Specify the daemon type (osd or mon) and its ID (numbers for OSDs, or short host names for Monitors) for example:

      [root@mon ~]# systemctl restart ceph-osd@1
      Copy to Clipboard Toggle word wrap
  2. Reproduce the failure, for example try to start the daemon again.
  3. Use the GNU Debugger (GDB) to generate a readable backtrace from an application core dump file:

    gdb /usr/bin/ceph-DAEMON /tmp/core.PID
    Copy to Clipboard Toggle word wrap

    Specify the daemon type and the PID of the failed process, for example:

    $ gdb /usr/bin/ceph-osd /tmp/core.123456
    Copy to Clipboard Toggle word wrap

    In the GDB command prompt disable paging and enable logging to a file by entering the commands set pag off and set log on:

    (gdb) set pag off
    (gdb) set log on
    Copy to Clipboard Toggle word wrap

    Apply the backtrace command to all threads of the process by entering thr a a bt full:

    (gdb) thr a a bt full
    Copy to Clipboard Toggle word wrap

    After the backtrace is generated turn off logging by entering set log off:

    (gdb) set log off
    Copy to Clipboard Toggle word wrap
  4. Transfer the log file gdb.txt to the system you access the Red Hat Customer Portal from and attach it to a support ticket.

9.2.3. Generating readable core dump files in containerized deployments

Follow this procedure to generate a core dump file if you use {storage-product} in containers. The procedure involves two scenarios of capturing the core dump file:

  • When a Ceph process terminates unexpectedly due to the SIGILL, SIGTRAP, SIGABRT, or SIGSEGV error.

or

  • Manually, for example for debugging issues such as Ceph processes are consuming high CPU cycles, or are not responding.

Prerequisites

  • Root-level access to the container node running the Ceph containers.
  • Installation of the appropriate debugging packages.
  • Installation of the GNU Project Debugger (gdb) package.

Procedure

  1. If a Ceph process terminates unexpectedly due to the SIGILL, SIGTRAP, SIGABRT, or SIGSEGV error:

    1. Set the core pattern to the systemd-coredump service on the node where the container with the failed Ceph process is running, for example:

      [root@mon]# echo "| /usr/lib/systemd/systemd-coredump %P %u %g %s %t %e" > /proc/sys/kernel/core_pattern
      Copy to Clipboard Toggle word wrap
    2. Watch for the next container failure due to a Ceph process and search for a core dump file in the /var/lib/systemd/coredump/ directory, for example:

      [root@mon]# ls -ltr /var/lib/systemd/coredump
      total 8232
      -rw-r-----. 1 root root 8427548 Jan 22 19:24 core.ceph-osd.167.5ede29340b6c4fe4845147f847514c12.15622.1584573794000000.xz
      Copy to Clipboard Toggle word wrap
  2. To manually capture a core dump file for the Ceph Monitors and Ceph Managers:

    1. Get the ceph-mon package details of the Ceph daemon from the container:

      [root@mon]# docker exec -it NAME /bin/bash
      [root@mon]# rpm -qa | grep ceph
      Copy to Clipboard Toggle word wrap

      Replace NAME with the name of the Ceph container.

    2. Make a backup copy and open for editing the ceph-mon@.service file:

      [root@mon]# cp /etc/systemd/system/ceph-mon@.service /etc/systemd/system/ceph-mon@.service.orig
      Copy to Clipboard Toggle word wrap
    3. In the ceph-mon@.service file, add these three options to the [Service] section, each on a separate line:

      --pid=host \
      --ipc=host \
      --cap-add=SYS_PTRACE \
      Copy to Clipboard Toggle word wrap

      Example

      [Unit]
      Description=Ceph Monitor
      After=docker.service
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/docker rm ceph-mon-%i
      ExecStartPre=/bin/sh -c '"$(command -v mkdir)" -p /etc/ceph /var/lib/ceph/mon'
      ExecStart=/usr/bin/docker run --rm --name ceph-mon-%i \
        --memory=924m \
      --cpu-quota=100000 \
      -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
      -v /etc/localtime:/etc/localtime:ro \
      --net=host \
      --privileged=true \
      --ipc=host \ 
      1
      
      --pid=host \ 
      2
      
      --cap-add=SYS_PTRACE \ 
      3
      
      -e IP_VERSION=4 \
              -e MON_IP=10.74.131.17 \
            -e CLUSTER=ceph \
        -e FSID=9448efca-b1a1-45a3-bf7b-b55cba696a6e \
        -e CEPH_PUBLIC_NETWORK=10.74.131.0/24 \
        -e CEPH_DAEMON=MON \
         \
        registry.access.redhat.com/rhceph/rhceph-3-rhel7:latest
      ExecStop=-/usr/bin/docker stop ceph-mon-%i
      ExecStopPost=-/bin/rm -f /var/run/ceph/ceph-mon.pd-cephcontainer-mon01.asok
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      
      [Install]
      WantedBy=multi-user.target
      Copy to Clipboard Toggle word wrap

    4. Restart the Ceph Monitor daemon:

      Syntax

      systemctl restart ceph-mon@MONITOR_ID
      Copy to Clipboard Toggle word wrap

      Replace MONITOR_ID with the ID number of the Ceph Monitor.

      Example

      [root@mon]# systemctl restart ceph-mon@1
      Copy to Clipboard Toggle word wrap

    5. Install the gdb package inside the Ceph Monitor container:

      [root@mon]# docker exec -it ceph-mon-MONITOR_ID /bin/bash
      sh $ yum install gdb
      Copy to Clipboard Toggle word wrap

      Replace MONITOR_ID with the ID number of the Ceph Monitor.

    6. Find the process ID:

      Syntax

      ps -aef | grep PROCESS | grep -v run
      Copy to Clipboard Toggle word wrap

      Replace PROCESS with the name of failed process, for example ceph-mon.

      Example

      [root@mon]# ps -aef | grep ceph-mon | grep -v run
      ceph       15390   15266  0 18:54 ?        00:00:29 /usr/bin/ceph-mon --cluster ceph --setroot ceph --setgroup ceph -d -i 5
      ceph       18110   17985  1 19:40 ?        00:00:08 /usr/bin/ceph-mon --cluster ceph --setroot ceph --setgroup ceph -d -i 2
      Copy to Clipboard Toggle word wrap

    7. Generate the core dump file:

      Syntax

      gcore ID
      Copy to Clipboard Toggle word wrap

      Replace ID with the ID of the failed process that you got from the previous step, for example 18110:

      Example

      [root@mon]# gcore 18110
      warning: target file /proc/18110/cmdline contained unexpected null characters
      Saved corefile core.18110
      Copy to Clipboard Toggle word wrap

    8. Verify that the core dump file has been generated correctly.

      Example

      [root@mon]# ls -ltr
      total 709772
      -rw-r--r--. 1 root root 726799544 Mar 18 19:46 core.18110
      Copy to Clipboard Toggle word wrap

    9. Copy the core dump file outside of the Ceph Monitor container:

      [root@mon]# docker cp ceph-mon-MONITOR_ID:/tmp/mon.core.MONITOR_PID /tmp
      Copy to Clipboard Toggle word wrap

      Replace MONITOR_ID with the ID number of the Ceph Monitor and replace MONITOR_PID with the process ID number.

    10. Restore the backup copy of the ceph-mon@.service file:

      [root@mon]# cp /etc/systemd/system/ceph-mon@.service.orig /etc/systemd/system/ceph-mon@.service
      Copy to Clipboard Toggle word wrap
    11. Restart the Ceph Monitor daemon:

      Syntax

      systemctl restart ceph-mon@MONITOR_ID
      Copy to Clipboard Toggle word wrap

      Replace MONITOR_ID with the ID number of the Ceph Monitor.

      Example

      [root@mon]# systemctl restart ceph-mon@1
      Copy to Clipboard Toggle word wrap

    12. Upload the core dump file for analysis by Red Hat support, see step 4.
  3. To manually capture a core dump file for Ceph OSDs:

    1. Get the ceph-osd package details of the Ceph daemon from the container:

      [root@osd]# docker exec -it NAME /bin/bash
      [root@osd]# rpm -qa | grep ceph
      Copy to Clipboard Toggle word wrap

      Replace NAME with the name of the Ceph container.

    2. Install the Ceph package for the same version of the ceph-osd package on the node where the Ceph containers are running:

      [root@osd]# yum install ceph-osd
      Copy to Clipboard Toggle word wrap

      If needed, enable the appropriate repository first. See the Enabling the Red Hat Ceph Storage repositories section in the Installation Guide for details.

    3. Find the ID of the process that has failed:

      ps -aef | grep PROCESS | grep -v run
      Copy to Clipboard Toggle word wrap

      Replace PROCESS with the name of failed process, for example ceph-osd.

      [root@osd]# ps -aef | grep ceph-osd | grep -v run
      ceph       15390   15266  0 18:54 ?        00:00:29 /usr/bin/ceph-osd --cluster ceph --setroot ceph --setgroup ceph -d -i 5
      ceph       18110   17985  1 19:40 ?        00:00:08 /usr/bin/ceph-osd --cluster ceph --setroot ceph --setgroup ceph -d -i 2
      Copy to Clipboard Toggle word wrap
    4. Generate the core dump file:

      gcore ID
      Copy to Clipboard Toggle word wrap

      Replace ID with the ID of the failed process that you got from the previous step, for example 18110:

      [root@osd]# gcore 18110
      warning: target file /proc/18110/cmdline contained unexpected null characters
      Saved corefile core.18110
      Copy to Clipboard Toggle word wrap
    5. Verify that the core dump file has been generated correctly.

      [root@osd]# ls -ltr
      total 709772
      -rw-r--r--. 1 root root 726799544 Mar 18 19:46 core.18110
      Copy to Clipboard Toggle word wrap
    6. Upload the core dump file for analysis by Red Hat support, see the next step.
  4. Upload the core dump file for analysis to a Red Hat support case. See Providing information to Red Hat Support engineers for details.

9.2.4. Additional Resources

맨 위로 이동
Red Hat logoGithubredditYoutubeTwitter

자세한 정보

평가판, 구매 및 판매

커뮤니티

Red Hat 문서 정보

Red Hat을 사용하는 고객은 신뢰할 수 있는 콘텐츠가 포함된 제품과 서비스를 통해 혁신하고 목표를 달성할 수 있습니다. 최신 업데이트를 확인하세요.

보다 포괄적 수용을 위한 오픈 소스 용어 교체

Red Hat은 코드, 문서, 웹 속성에서 문제가 있는 언어를 교체하기 위해 최선을 다하고 있습니다. 자세한 내용은 다음을 참조하세요.Red Hat 블로그.

Red Hat 소개

Red Hat은 기업이 핵심 데이터 센터에서 네트워크 에지에 이르기까지 플랫폼과 환경 전반에서 더 쉽게 작업할 수 있도록 강화된 솔루션을 제공합니다.

Theme

© 2025 Red Hat