3.4.3. Inspecting application crash states with core dumps

Procedure

1. To identify the executable file where the crash occurred, run the eu-unstrip command with the core dump file:

   $ eu-unstrip -n --core=./core.9814

   0x400000+0x207000 2818b2009547f780a5639c904cded443e564973e@0x400284 /usr/bin/sleep /usr/lib/debug/bin/sleep.debug [exe]
   0x7fff26fff000+0x1000 1e2a683b7d877576970e4275d41a6aaec280795e@0x7fff26fff340 . - linux-vdso.so.1
   0x35e7e00000+0x3b6000 374add1ead31ccb449779bc7ee7877de3377e5ad@0x35e7e00280 /usr/lib64/libc-2.14.90.so /usr/lib/debug/lib64/libc-2.14.90.so.debug libc.so.6
   0x35e7a00000+0x224000 3ed9e61c2b7e707ce244816335776afa2ad0307d@0x35e7a001d8 /usr/lib64/ld-2.14.90.so /usr/lib/debug/lib64/ld-2.14.90.so.debug ld-linux-x86-64.so.2

   The output contains details for each module on a line, separated by spaces. The information is listed in this order:

   1. The memory address where the module was mapped
   2. The build-id of the module and where in the memory it was found
   3. The module’s executable file name, displayed as - when unknown, or as . when the module has not been loaded from a file
   4. The source of debugging information, displayed as a file name when available, as . when contained in the executable file itself, or as - when not present at all
   5. The shared library name (soname) or [exe] for the main module

   In this example, the important details are the file name /usr/bin/sleep and the build-id 2818b2009547f780a5639c904cded443e564973e on the line containing the text [exe]. With this information, you can identify the executable file required for analyzing the core dump.

2. Get the executable file that crashed.

   • If possible, copy it from the system where the crash occurred. Use the file name extracted from the core file.

   • You can also use an identical executable file on your system. Each executable file built on Red Hat Enterprise Linux contains a note with a unique build-id value. Determine the build-id of the relevant locally available executable files:

     $ eu-readelf -n executable_file

     Use this information to match the executable file on the remote system with your local copy. The build-id of the local file and build-id listed in the core dump must match.

   • Finally, if the application is installed from an RPM package, you can get the executable file from the package. Use the sosreport output to find the exact version of the package required.
3. Get the shared libraries used by the executable file. Use the same steps as for the executable file.
4. If the application is distributed as a package, load the executable file in GDB, to display hints for missing debuginfo packages. For more details, see Getting debuginfo packages for an application or library using GDB.

5. To examine the core file in detail, load the executable file and core dump file with GDB:

   $ gdb -e executable_file -c core_file

   Further messages about missing files and debugging information help you identify what is missing for the debugging session. Return to the previous step if needed.

   If the application’s debugging information is available as a file instead of as a package, load this file in GDB with the symbol-file command:

   (gdb) symbol-file program.debug

   Replace program.debug with the actual file name.

   注意

   It might not be necessary to install the debugging information for all executable files contained in the core dump. Most of these executable files are libraries used by the application code. These libraries might not directly contribute to the problem you are analyzing, and you do not need to include debugging information for them.

6. Use the GDB commands to inspect the state of the application at the moment it crashed. See Inspecting application internal state with GDB.

   注意

   When analyzing a core file, GDB is not attached to a running process. Commands for controlling execution have no effect.

7. To see only the stack of the application at the moment it terminated, open the core file with the eu-stack utility:

   $ eu-stack --core core-file

   This will display listing of the application’s stack.