4.2. Installing Debuginfo Packages
Red Hat Enterprise Linux also provides
-debuginfo packages for all architecture-dependent RPMs included in the operating system. A packagename-debuginfo-version-release.architecture.rpm package contains detailed information about the relationship of the package source files and the final installed binary. The debuginfo packages contain both .debug files, which in turn contain DWARF debuginfo and the source files used for compiling the binary packages.
Note
Most of the debugger functionality is missed if attempting to debug a package without having its debuginfo equivalent installed. For example, the names of exported shared library functions will still be available, but the matching source file lines will not be without the debuginfo package installed.
Use
gcc compilation option -g for your own programs. The debugging experience is better if no optimizations (gcc option -O, such as -O2) is applied with -g.
For Red Hat Enterprise Linux 6, the debuginfo packages are now available on a new channel on the Red Hat Network. To install the
-debuginfo package of a package (that is, typically packagename-debuginfo), first the machine has to be subscribed to the corresponding Debuginfo channel. For example, for Red Hat Enterprise Server 6, the corresponding channel would be Red Hat Enterprise Linux Server Debuginfo (v. 6).
Red Hat Enterprise Linux system packages are compiled with optimizations (gcc option
-O2). This means that some variables will be displayed as <optimized out>. Stepping through code will 'jump' a little but a crash can still be analyzed. If some debugging information is missing because of the optimizations, the right variable information can be found by disassembling the code and matching it to the source manually. This is applicable only in exceptional cases and is not suitable for regular debugging.
For system packages, GDB informs the user if it is missing some debuginfo packages that limit its functionality.
gdb ls [...] Reading symbols from /bin/ls...(no debugging symbols found)...done. Missing separate debuginfos, use: debuginfo-install coreutils-8.4-16.el6.x86_64 (gdb) q
If the system package to be debugged is known, use the command suggested by GDB above. It will also automatically install all the debug packages packagename depends on.
# debuginfo-install packagename
4.2.1. Installing Debuginfo Packages for Core Files Analysis
A core file is a representation of the memory image at the time of a process crash. For bug reporting of system program crashes, Red Hat recommends the use of the ABRT tool, explained in the Automatic Bug Reporting Tool chapter in the Red Hat Deployment Guide. If ABRT is not suitable for your purposes, the steps it automates are explained here.
If the
ulimit -c unlimited setting is in use when a process crashes, the core file is dumped into the current directory. The core file contains only the memory areas modified by the process from the original state of disk files. In order to perform a full analysis of a crash, a core file is required to have:
- the core file itself
- the executable binary which has crashed, such as
/usr/sbin/sendmail - all the shared libraries loaded in the binary when it crashed
- .debug files and source files (both stored in debuginfo RPMs) for the executable and all of its loaded libraries
For a proper analysis, either the exact
version-release.architecture for all the RPMs involved or the same build of your own compiled binaries is needed. At the time of the crash, the application may have already recompiled or been updated by yum on the disk, rendering the files inappropriate for the core file analysis.
The core file contains build-ids of all the binaries involved. For more information on build-id, see Section 3.3, “build-id Unique Identification of Binaries”. The contents of the core file can be displayed by:
$ eu-unstrip -n --core=./core.9814 0x400000+0x207000 2818b2009547f780a5639c904cded443e564973e@0x400284 /bin/sleep /usr/lib/debug/bin/sleep.debug [exe] 0x7fff26fff000+0x1000 1e2a683b7d877576970e4275d41a6aaec280795e@0x7fff26fff340 . - linux-vdso.so.1 0x35e7e00000+0x3b6000 374add1ead31ccb449779bc7ee7877de3377e5ad@0x35e7e00280 /lib64/libc-2.14.90.so /usr/lib/debug/lib64/libc-2.14.90.so.debug libc.so.6 0x35e7a00000+0x224000 3ed9e61c2b7e707ce244816335776afa2ad0307d@0x35e7a001d8 /lib64/ld-2.14.90.so /usr/lib/debug/lib64/ld-2.14.90.so.debug ld-linux-x86-64.so.2
The meaning of the columns in each line are:
- The in-memory address where the specific binary was mapped to (for example,
0x400000in the first line). - The size of the binary (for example,
+0x207000in the first line). - The 160-bit SHA-1 build-id of the binary (for example,
2818b2009547f780a5639c904cded443e564973ein the first line). - The in-memory address where the build-id bytes were stored (for example,
@0x400284in the first line). - The on-disk binary file, if available (for example,
/bin/sleepin the first line). This was found byeu-unstripfor this module. - The on-disk debuginfo file, if available (for example,
/usr/lib/debug/bin/sleep.debug). However, best practice is to use the binary file reference instead. - The shared library name as stored in the shared library list in the core file (for example,
libc.so.6in the third line).
For each build-id (for example,
ab/cdef0123456789012345678901234567890123) a symbolic link is included in its debuginfo RPM. Using the /bin/sleep executable above as an example, the coreutils-debuginfo RPM contains, among other files:
lrwxrwxrwx 1 root root 24 Nov 29 17:07 /usr/lib/debug/.build-id/28/18b2009547f780a5639c904cded443e564973e -> ../../../../../bin/sleep* lrwxrwxrwx 1 root root 21 Nov 29 17:07 /usr/lib/debug/.build-id/28/18b2009547f780a5639c904cded443e564973e.debug -> ../../bin/sleep.debug
In some cases (such as loading a core file), GDB does not know the name, version, or release of a
name-debuginfo-version-release.rpm package; it only knows the build-id. In such cases, GDB suggests a different command:
gdb -c ./core [...] Missing separate debuginfo for the main executable filename Try: yum --disablerepo='*' --enablerepo='*debug*' install /usr/lib/debug/.build-id/ef/dd0b5e69b0742fa5e5bad0771df4d1df2459d1
The version-release.architecture of the binary package packagename-debuginfo-version-release.architecture.rpm must be an exact match. If it differs then GDB cannot use the debuginfo package. Even the same version-release.architecture from a different build leads to an incompatible debuginfo package. If GDB reports a missing debuginfo, ensure to recheck:
rpm -q packagename packagename-debuginfo- The version-release.architecture definitions should match.
rpm -V packagename packagename-debuginfo- This command should produce no output, except possibly modified configuration files of packagename, for example.
rpm -qi packagename packagename-debuginfo- The version-release.architecture should display matching information for Vendor, Build Date, and Build Host. For example, using a CentOS debuginfo RPM for a Red Hat Enterprise Linux RPM package will not work.
If the required build-id is known, the following command can query which RPM contains it:
$ repoquery --disablerepo='*' --enablerepo='*-debug*' -qf /usr/lib/debug/.build-id/ef/dd0b5e69b0742fa5e5bad0771df4d1df2459d1
For example, a version of an executable which matches the core file can be installed by:
# yum --enablerepo='*-debug*' install $(eu-unstrip -n --core=./core.9814 | sed -e 's#^[^ ]* \(..\)\([^@ ]*\).*$#/usr/lib/debug/.build-id/\1/\2#p' -e 's/$/.debug/')
Similar methods are available if the binaries are not packaged into RPMs and stored in yum repositories. It is possible to create local repositories with custom application builds by using
/usr/bin/createrepo.
