Debug a crashed application

Sometimes, it is not possible to debug an application directly. In these situations, you can collect information about the application at the moment of its termination and analyze it afterwards.

Core dumps: what they are and how to use them
Copy link

A core dump records parts of an application’s memory when the application stops. After an application fails, you can analyze the core dump, along with the executable and debuginfo, by using a debugger.

The Linux operating system kernel can record core dumps automatically, if this functionality is enabled. Alternatively, send a signal to any running application to generate a core dump regardless of its actual state.

Warning

Some limits might affect the ability to generate a core dump. To see the current limits:

$ ulimit -a

Record application crashes with core dumps
Copy link

To record application crashes, configure core dump saving and add information about the system.

Procedure
Copy link

  1. To enable core dumps, ensure that the /etc/systemd/system.conf file contains the following lines: 
    DumpCore=yes
DefaultLimitCORE=infinity

    You can also add comments describing if these settings were previously present, and what the previous values were. This will enable you to reverse these changes later, if needed. Comments are lines starting with the # character.

    Changing the file requires administrator level access.

  2. Apply the new configuration: 
    # systemctl daemon-reexec
  3. Remove the limits for core dump sizes: 
    # ulimit -c unlimited

    To reverse this change, run the command with value 0 instead of unlimited.

  4. Install the sos package which provides the sosreport utility for collecting system information: 
    # dnf install sos
  5. When an application crashes, a core dump is generated and handled by systemd-coredump.
  6. Create an SOS report to provide additional information about the system: 
    # sosreport

    This creates a .tar archive containing information about your system, such as copies of configuration files.

  7. Locate the core dump: 
    $ coredumpctl list executable-name
  8. Export the core dump: 
    $ coredumpctl dump executable-name > /path/to/file-for-export

    If the application crashed multiple times, output of the first command lists more captured core dumps. In that case, construct for the second command a more precise query by using the other information. See the coredumpctl(1) manual page for details.

  9. Transfer the core dump and the SOS report to the computer where the debugging will take place. Transfer the executable file, too, if it is known.
    Important
    When the executable file is not known, subsequent analysis of the core file identifies it.
  10. Optional: Remove the core dump and SOS report after transferring them, to free up disk space.

Inspect application crash states with core dumps
Copy link

To inspect the state of an application at the moment it terminated unexpectedly, use core dumps.

Before you begin
Copy link

  • You must have a core dump file and sosreport from the system where the crash occurred.
  • GDB and elfutils must be installed on your system.

Procedure
Copy link

  1. To identify the executable file where the crash occurred, run the eu-unstrip command with the core dump file: 
    $ eu-unstrip -n --core=./core.9814
     
    0x400000+0x207000 2818b2009547f780a5639c904cded443e564973e@0x400284 /usr/bin/sleep /usr/lib/debug/bin/sleep.debug [exe]
0x7fff26fff000+0x1000 1e2a683b7d877576970e4275d41a6aaec280795e@0x7fff26fff340 . - linux-vdso.so.1
0x35e7e00000+0x3b6000 374add1ead31ccb449779bc7ee7877de3377e5ad@0x35e7e00280 /usr/lib64/libc-2.14.90.so /usr/lib/debug/lib64/libc-2.14.90.so.debug libc.so.6
0x35e7a00000+0x224000 3ed9e61c2b7e707ce244816335776afa2ad0307d@0x35e7a001d8 /usr/lib64/ld-2.14.90.so /usr/lib/debug/lib64/ld-2.14.90.so.debug ld-linux-x86-64.so.2

    The output contains details for each module on a line, separated by spaces. The information is listed in this order:

    1. The memory address where the module was mapped
    2. The build-id of the module and where in the memory it was found
    3. The module’s executable file name, displayed as - when unknown, or as . when the module has not been loaded from a file
    4. The source of debugging information, displayed as a file name when available, as . when contained in the executable file itself, or as - when not present at all
    5. The shared library name (soname) or [exe] for the main module

    In this example, the important details are the file name /usr/bin/sleep and the build-id 2818b2009547f780a5639c904cded443e564973e on the line containing the text [exe]. With this information, you can identify the executable file required for analyzing the core dump.

  2. Get the executable file that crashed.
    • If possible, copy it from the system where the crash occurred. Use the file name extracted from the core file.
    • You can also use an identical executable file on your system. Each executable file built on Red Hat Enterprise Linux contains a note with a unique build-id value. Determine the build-id of the relevant locally available executable files: 
      $ eu-readelf -n executable_file

      Use this information to match the executable file on the remote system with your local copy. The build-id of the local file and build-id listed in the core dump must match.

    • Finally, if the application is installed from an RPM package, you can get the executable file from the package. Use the sosreport output to find the exact version of the package required.
  3. Get the shared libraries used by the executable file. Use the same steps as for the executable file.
  4. If the application is distributed as a package, load the executable file in GDB, to display hints for missing debuginfo packages. For more details, see Getting debuginfo packages for an application or library using GDB.
  5. To examine the core file in detail, load the executable file and core dump file with GDB: 
    $ gdb -e executable_file -c core_file

    Further messages about missing files and debugging information help you identify what is missing for the debugging session. Return to the previous step if needed.

    If the application’s debugging information is available as a file instead of as a package, load this file in GDB with the symbol-file command:

    
            (gdb) symbol-file program.debug

    Replace program.debug with the actual file name.

    Note
    It might not be necessary to install the debugging information for all executable files contained in the core dump. Most of these executable files are libraries used by the application code. These libraries might not directly contribute to the problem you are analyzing, and you do not need to include debugging information for them.
  6. Use the GDB commands to inspect the state of the application at the moment it crashed. See Inspecting application internal state with GDB.
    Note
    When analyzing a core file, GDB is not attached to a running process. Commands for controlling execution have no effect.
  7. To see only the stack of the application at the moment it terminated, open the core file with the eu-stack utility: 
    $ eu-stack --core core-file

    This will display listing of the application’s stack.

Create and access a core dump with coredumpctl
Copy link

To manage and analyze core dumps directly on the affected system, use coredumpctl. This tool simplifies finding, capturing, and inspecting crash data. Identify an unresponsive process, force a core dump, and verify its successful capture to diagnose application failures.

Before you begin
Copy link

  • The system must be configured to use systemd-coredump for core dump handling. To verify this is true: 
    $ sysctl kernel.core_pattern

    The configuration is correct if the output starts with the following:

    kernel.core_pattern = |/usr/lib/systemd/systemd-coredump

Procedure
Copy link

  1. Find the PID of the hung process, based on a known part of the executable file name: 
    $ pgrep -a executable-name-fragment

    This command will output a line in the form

    PID command-line

    Use the command-line value to verify that the PID belongs to the intended process.

    For example:

    $ pgrep -a bc
     
    5459 bc
  2. Send an abort signal to the process: 
    # kill -ABRT PID
  3. Verify that the core has been captured by coredumpctl: 
    $ coredumpctl list PID

    For example:

    $ coredumpctl list 5459
     
    TIME                            PID   UID   GID SIG COREFILE  EXE
Thu 2019-11-07 15:14:46 CET    5459  1000  1000   6 present   /usr/bin/bc
  4. Further examine or use the core file as needed.

    You can specify the core dump by PID and other values. See the coredumpctl(1) manual page for further details.

    Next steps

    • To show details of the core file, run: 
      $ coredumpctl info PID
    • To load the core file in the GDB debugger, run: 
      $ coredumpctl debug PID

      Depending on the availability of debugging information, GDB might suggest commands to run, such as:

      Missing separate debuginfos, use: dnf debuginfo-install bc-1.07.1-23.el10.x86_64

      For more details on this process, see Getting debuginfo packages for an application or library using GDB.

    • To export the core file for further processing elsewhere, run: 
      $ coredumpctl dump PID > /path/to/file_for_export
    • Replace /path/to/file_for_export with the file where you want to put the core dump. 
      $ coredumpctl dump PID > /path/to/file_for_export

Dump process memory with gcore
Copy link

To capture the memory state of a running process without terminating it, use the gcore utility. This creates a core dump file for offline analysis. The result is a snapshot of the application’s memory that helps you investigate issues while the service remains available.

Before you begin
Copy link

  • You must understand what core dumps are and how they are created. For details, see Core dumps: what they are and how to use them.
  • GDB must be installed on the system. For details, see Installing tools for debugging applications. .Procedure
    1. Identify the process id (pid). Use tools such as ps, pgrep, and top: 
      $ ps -C some-program
    2. Dump the memory of this process: 
      $ gcore -o filenamepid

      This creates a file filename and dumps the process memory in it. While the memory is being dumped, the execution of the process is halted.

    3. After the core dump is finished, the process resumes normal execution.
    4. Create an SOS report to provide additional information about the system: 
      # sosreport

      This creates a tar archive containing information about your system, such as copies of configuration files.

    5. Transfer the program’s executable file, core dump, and the SOS report to the computer where the debugging will take place.
    6. Optional: Remove the core dump and SOS report after transferring them, to free up disk space.

Dump protected process memory with GDB
Copy link

To dump protected process memory, configure GNU Debugger (GDB) to ignore core dump filters. Capture memory regions flagged as non-dumpable, such as those conserving resources or holding sensitive data. Use the gcore command within GDB to generate the complete core file.

Before you begin
Copy link

  • You must understand what core dumps are. For details, see Core dumps: what they are and how to use them.
  • GDB must be installed on the system. For details, see Installing tools for debugging applications.
  • GDB must be already attached to the process with protected memory. For details, see Attaching GDB to a process. .Procedure
    1. Set GDB to ignore the settings in the /proc/PID/coredump_filter file: 
      (gdb) set use-coredump-filter off
    2. Set GDB to ignore the memory page flag VM_DONTDUMP: 
      (gdb) set dump-excluded-mappings on
    3. Dump the memory: 
      (gdb) gcore core-file

      Replace core-file with name of file where you want to dump the memory.