Chapter 3. Debugging Applications

Procedure

1. Compile and link your code with the -g option to generate debugging information:

   $ gcc ... -g ...

2. Optional: Set the optimization level to -Og:

   $ gcc ... -g -Og ...

   Compiler optimizations can make executable code hard to relate to the source code. The -Og option optimizes the code without interfering with debugging. However, be aware that changing optimization levels can alter the program’s behavior.

3. Optional: Use -g for moderate debugging information, or -g3 to include macro definitions:

   $ gcc ... -g3 ...

Procedure

1. Start GDB attached to the application or library you want to debug. GDB automatically recognizes missing debugging information and suggests a command to run.

   $ gdb -q /bin/ls

   Reading symbols from /bin/ls...Reading symbols from .gnu_debugdata for /usr/bin/ls...(no debugging symbols found)...done.
   (no debugging symbols found)...done.
   Missing separate debuginfos, use: dnf debuginfo-install coreutils-9.5-6.el10.x86_64
   (gdb)

2. Exit GDB: type q and confirm with Enter.

   (gdb) q

3. Run the command suggested by GDB to install the required debuginfo packages:

   # dnf debuginfo-install coreutils-9.5-6.el10.x86_64

   The dnf package management tool provides a summary of the changes, asks for confirmation and once you confirm, downloads and installs all the necessary files.

4. In case GDB is not able to suggest the debuginfo package, follow the procedure described in

   • Getting debuginfo packages for an application or library manually.

Procedure

1. Find the executable file of the application or library.

   1. Use the which command to find the application file.

      $ which less

      /usr/bin/less

   2. Use the locate command to find the library file.

      $ locate libz | grep so

      /usr/lib64/libz.so.1
      /usr/lib64/libz.so.1.2.11

      If the original reasons for debugging include error messages, pick the result where the library has the same additional numbers in its file name as those mentioned in the error messages. If in doubt, try following the rest of the procedure with the result where the library file name includes no additional numbers.

      Note

      The locate command is provided by the mlocate package. To install it and enable its use:

      # dnf install mlocate

      1. Update the database:

         # updatedb

2. Search for a name and version of the package that provided the file:

   $ rpm -qf /usr/lib64/libz.so.1.3.1.zlib-ng

   zlib-ng-compat-2.2.3-1.el10.x86_64

   The output provides details for the installed package in the name:epoch-version.release.architecture format.

   Important

   If this step does not produce any results, it is not possible to determine which package provided the binary file. There are several possible cases:

   • The file is installed from a package which is not known to package management tools in their current configuration.
   • The file is installed from a locally downloaded and manually installed package. Determining a suitable debuginfo package automatically is impossible in that case.
   • Your package management tools are misconfigured.
   • The file is not installed from any package. In such a case, no corresponding debuginfo package exists.

   Because further steps depend on this one, you must resolve this situation or stop this procedure. Describing the exact troubleshooting steps is beyond the scope of this procedure.

3. Install the debuginfo packages using the dnf debuginfo-install utility. In the command, use the package name and other details you determined during the previous step:

   # dnf debuginfo-install zlib-ng-compat-2.2.3-1.el10.x86_64

Procedure

1. Identify the system calls to monitor.

   • Start strace and attach it to the program.

     • If the program you want to monitor is not running, start strace and specify the program:

       $ strace -fvttTyy -s 256 -e trace=call program

     • If the program is already running, find its process id (pid):

       $ ps -C program

     • Attach strace to the process:

       $ strace -fvttTyy -s 256 -e trace=call -ppid

   • Replace call with the system calls to be displayed. You can use the -e trace=call option multiple times. If left out, strace will display all system call types. See the strace(1) manual page for more information.
   • If you do not want to trace any forked processes or threads, omit the -f option.

2. The strace tool displays the system calls made by the application and their details.

   In most cases, an application and its libraries make a large number of calls and strace output displays immediately, if no filter for system calls is set.

3. The strace tool exits when the program exits.

   To terminate the monitoring before the traced program exits, press Ctrl+C.

   • If strace started the program, the program terminates together with strace.
   • If you attached strace to an already running program, the program terminates together with strace.

4. Analyze the list of system calls done by the application.

   • Problems with resource access or availability are present in the log as calls returning errors.
   • Values passed to the system calls and patterns of call sequences provide insight into the causes of the application’s behaviour.
   • If the application crashes, the important information is probably at the end of log.

   • The output contains a large amount of unnecessary information. However, you can construct a more precise filter for the system calls of interest and repeat the procedure.

     Note

     It is advantageous to both see the output and save it to a file. Use the tee command to achieve this:

     $ strace ... |& tee your_log_file.log

Procedure

1. Identify the libraries and functions of interest, if possible.

2. Start ltrace and attach it to the program.

   Note

   In Red Hat Enterprise Linux 10, a known issue prevents ltrace from tracing system executable files. This limitation does not apply to executable files built by users.

   • If the program you want to monitor is not running, start ltrace and specify program:

     $ ltrace -f -l library -e function program

   • If the program is already running, find its process id (pid):

     $ ps -C program

   • Attach ltrace to the process:

     $ ltrace -f -l library -e function -ppid program

   • Use the -e, -f and -l options to filter the output:

     • Supply the function names to be displayed as function. The -e function option can be used multiple times. If left out, ltrace displays calls to all functions.
     • Instead of specifying functions, you can specify whole libraries with the -l library option. This option behaves similarly to the -e function option.
     • If you do not want to trace any forked processes or threads, omit the -f option.

     See the ltrace(1)_ manual page for more information.

3. ltrace displays the library calls made by the application.

   In most cases, an application makes a large number of calls and ltrace output displays immediately, if no filter is set.

4. ltrace exits when the program exits.

   To terminate the monitoring before the traced program exits, press ctrl+C.

   • If ltrace started the program, the program terminates together with ltrace.
   • If you attached ltrace to an already running program, the program terminates together with ltrace.
5. Stop ltrace by pressing Ctrl+C.

6. Analyze the list of library calls done by the application.

   • If the application crashes, the important information is probably at the end of log.

   • The output contains a large amount of unnecessary information. However, you can construct a more precise filter and repeat the procedure.

     Note

     It is advantageous to both see the output and save it to a file. Use the tee command to achieve this:

     $ ltrace ... |& tee your_log_file.log

Procedure

1. Find the process ID (pid) of the process you want to monitor:

   $ ps -aux

2. Run SystemTap with the strace.stp script:

   # stap /usr/share/systemtap/examples/process/strace.stp -x pid

   The value of pid is the process id.

   The script is compiled to a kernel module, which is then loaded. This introduces a slight delay between entering the command and getting the output.

3. When the process performs a system call, the call name and its parameters are printed to the terminal.
4. The script exits when the process terminates, or when you press Ctrl+C.

Procedure

1. Set the catchpoint:

   (gdb) catch syscall syscall-name

   The command catch syscall sets a special type of breakpoint that halts execution when the program performs a system call.

   The syscall-name option specifies the name of the call. You can specify multiple catchpoints for various system calls. Leaving out the syscall-name option causes GDB to stop on any system call.

2. Start execution of the program.

   • If the program has not started execution, start it:

     (gdb) r

   • If the program execution is halted, resume it:

     (gdb) c

3. GDB halts execution after the program performs any specified system call.
4. Use further GDB commands to examine the program state and advance execution, according to the particular situation.

5. To exit the GDB debugging session:

   (gdb) q

Procedure

1. Set the catchpoint:

   (gdb) catch signal signal-type

   The command catch signal sets a special type of a breakpoint that halts execution when a signal is received by the program. The signal-type option specifies the type of the signal. Use the special value 'all' to catch all signals.

2. Let the program run.

   • If the program has not started execution, start it:

     (gdb) r

   • If the program execution is halted, resume it:

     (gdb) c

3. GDB halts execution after the program receives any specified signal.
4. Continue by debugging the program code handling the signal, or resume execution with the knowledge of the signal being received.

5. Later, to exit the GDB debugging session:

   (gdb) q

Procedure

1. To enable core dumps, ensure that the /etc/systemd/system.conf file contains the following lines:

   DumpCore=yes
   DefaultLimitCORE=infinity

   You can also add comments describing if these settings were previously present, and what the previous values were. This will enable you to reverse these changes later, if needed. Comments are lines starting with the # character.

   Changing the file requires administrator level access.

2. Apply the new configuration:

   # systemctl daemon-reexec

3. Remove the limits for core dump sizes:

   # ulimit -c unlimited

   To reverse this change, run the command with value 0 instead of unlimited.

4. Install the sos package which provides the sosreport utility for collecting system information:

   # dnf install sos

5. When an application crashes, a core dump is generated and handled by systemd-coredump.

6. Create an SOS report to provide additional information about the system:

   # sosreport

   This creates a .tar archive containing information about your system, such as copies of configuration files.

7. Locate the core dump:

   $ coredumpctl list executable-name

8. Export the core dump:

   $ coredumpctl dump executable-name > /path/to/file-for-export

   If the application crashed multiple times, output of the first command lists more captured core dumps. In that case, construct for the second command a more precise query by using the other information. See the coredumpctl(1) manual page for details.

9. Transfer the core dump and the SOS report to the computer where the debugging will take place. Transfer the executable file, too, if it is known.

   Important

   When the executable file is not known, subsequent analysis of the core file identifies it.

10. Optional: Remove the core dump and SOS report after transferring them, to free up disk space.

Procedure

1. To identify the executable file where the crash occurred, run the eu-unstrip command with the core dump file:

   $ eu-unstrip -n --core=./core.9814

   0x400000+0x207000 2818b2009547f780a5639c904cded443e564973e@0x400284 /usr/bin/sleep /usr/lib/debug/bin/sleep.debug [exe]
   0x7fff26fff000+0x1000 1e2a683b7d877576970e4275d41a6aaec280795e@0x7fff26fff340 . - linux-vdso.so.1
   0x35e7e00000+0x3b6000 374add1ead31ccb449779bc7ee7877de3377e5ad@0x35e7e00280 /usr/lib64/libc-2.14.90.so /usr/lib/debug/lib64/libc-2.14.90.so.debug libc.so.6
   0x35e7a00000+0x224000 3ed9e61c2b7e707ce244816335776afa2ad0307d@0x35e7a001d8 /usr/lib64/ld-2.14.90.so /usr/lib/debug/lib64/ld-2.14.90.so.debug ld-linux-x86-64.so.2

   The output contains details for each module on a line, separated by spaces. The information is listed in this order:

   1. The memory address where the module was mapped
   2. The build-id of the module and where in the memory it was found
   3. The module’s executable file name, displayed as - when unknown, or as . when the module has not been loaded from a file
   4. The source of debugging information, displayed as a file name when available, as . when contained in the executable file itself, or as - when not present at all
   5. The shared library name (soname) or [exe] for the main module

   In this example, the important details are the file name /usr/bin/sleep and the build-id 2818b2009547f780a5639c904cded443e564973e on the line containing the text [exe]. With this information, you can identify the executable file required for analyzing the core dump.

2. Get the executable file that crashed.

   • If possible, copy it from the system where the crash occurred. Use the file name extracted from the core file.

   • You can also use an identical executable file on your system. Each executable file built on Red Hat Enterprise Linux contains a note with a unique build-id value. Determine the build-id of the relevant locally available executable files:

     $ eu-readelf -n executable_file

     Use this information to match the executable file on the remote system with your local copy. The build-id of the local file and build-id listed in the core dump must match.

   • Finally, if the application is installed from an RPM package, you can get the executable file from the package. Use the sosreport output to find the exact version of the package required.
3. Get the shared libraries used by the executable file. Use the same steps as for the executable file.
4. If the application is distributed as a package, load the executable file in GDB, to display hints for missing debuginfo packages. For more details, see Getting debuginfo packages for an application or library using GDB.

5. To examine the core file in detail, load the executable file and core dump file with GDB:

   $ gdb -e executable_file -c core_file

   Further messages about missing files and debugging information help you identify what is missing for the debugging session. Return to the previous step if needed.

   If the application’s debugging information is available as a file instead of as a package, load this file in GDB with the symbol-file command:

   (gdb) symbol-file program.debug

   Replace program.debug with the actual file name.

   Note

   It might not be necessary to install the debugging information for all executable files contained in the core dump. Most of these executable files are libraries used by the application code. These libraries might not directly contribute to the problem you are analyzing, and you do not need to include debugging information for them.

6. Use the GDB commands to inspect the state of the application at the moment it crashed. See Inspecting Application Internal State with GDB.

   Note

   When analyzing a core file, GDB is not attached to a running process. Commands for controlling execution have no effect.

7. To see only the stack of the application at the moment it terminated, open the core file with the eu-stack utility:

   $ eu-stack --core core-file

   This will display listing of the application’s stack.

Procedure

1. Find the PID of the hung process, based on a known part of the executable file name:

   $ pgrep -a executable-name-fragment

   This command will output a line in the form

   PID command-line

   Use the command-line value to verify that the PID belongs to the intended process.

   For example:

   $ pgrep -a bc

   5459 bc

2. Send an abort signal to the process:

   # kill -ABRT PID

3. Verify that the core has been captured by coredumpctl:

   $ coredumpctl list PID

   For example:

   $ coredumpctl list 5459

   TIME                            PID   UID   GID SIG COREFILE  EXE
   Thu 2019-11-07 15:14:46 CET    5459  1000  1000   6 present   /usr/bin/bc

4. Further examine or use the core file as needed.

   You can specify the core dump by PID and other values. See the coredumpctl(1) manual page for further details.

   Next steps

   • To show details of the core file, run:

     $ coredumpctl info PID

   • To load the core file in the GDB debugger, run:

     $ coredumpctl debug PID

     Depending on the availability of debugging information, GDB might suggest commands to run, such as:

     Missing separate debuginfos, use: dnf debuginfo-install bc-1.07.1-23.el10.x86_64

     For more details on this process, see Getting debuginfo packages for an application or library using GDB.

   • To export the core file for further processing elsewhere, run:

     $ coredumpctl dump PID > /path/to/file_for_export

   • Replace /path/to/file_for_export with the file where you want to put the core dump.

     $ coredumpctl dump PID > /path/to/file_for_export

Procedure

1. Identify the process id (pid). Use tools such as ps, pgrep, and top:

   $ ps -C some-program

2. Dump the memory of this process:

   $ gcore -o filename pid

   This creates a file filename and dumps the process memory in it. While the memory is being dumped, the execution of the process is halted.

3. After the core dump is finished, the process resumes normal execution.

4. Create an SOS report to provide additional information about the system:

   # sosreport

   This creates a tar archive containing information about your system, such as copies of configuration files.

5. Transfer the program’s executable file, core dump, and the SOS report to the computer where the debugging will take place.
6. Optional: Remove the core dump and SOS report after transferring them, to free up disk space.

Procedure

1. Set GDB to ignore the settings in the /proc/PID/coredump_filter file:

   (gdb) set use-coredump-filter off

2. Set GDB to ignore the memory page flag VM_DONTDUMP:

   (gdb) set dump-excluded-mappings on

3. Dump the memory:

   (gdb) gcore core-file

   Replace core-file with name of file where you want to dump the memory.