Inspect application internal state with GDB

To identify why an application fails, you can control its execution and examine its internal state with the GNU Debugger (GDB).

GNU debugger (GDB)
Copy link

Use the GNU Debugger (GDB) to inspect program execution and post-crash states. Also, you can analyze internal data and control execution flow when tracking down runtime errors. This command-line tool shows the detailed application state needed to identify and fix bugs in complex applications.

GDB capabilities

A single GDB session can debug the following types of programs:

  • Multithreaded and forking programs
  • Multiple programs at once
  • Programs on remote machines or in containers with the gdbserver utility connected over a TCP/IP network connection
Debugging requirements

To debug any executable code, GDB requires debugging information for that particular code:

  • For programs developed by you, you can create the debugging information while building the code.
  • For system programs installed from packages, you must install their debuginfo packages.

Attach GDB to a process
Copy link

To examine a system process, attach the GNU Debugger (GDB) to the process.

Before you begin
Copy link

  • GDB must be installed on the system. For details, see Installing tools for debugging applications. .Procedure
  • Start a program with GDB.
    1. Launch a program using GDB: 
      $ gdb program

      Replace program with a file name or path to the program.

      GDB is ready to start execution of the program. You can configure breakpoints and the gdb environment before beginning the execution of the process with the run command. For more information on setting breakpoints, see Using GDB breakpoints to stop execution at defined code locations.

    2. Attach GDB to an already running process.
    3. Find the process ID (pid) with the ps command: 
      $ ps -C program -o pid h
       
       pid

      Replace program with a file name or path to the program.

    4. Attach GDB to this process: 
      $ gdb -p pid

      Replace pid with an actual process ID number from the ps output.

  • Attach an active GDB session to a running program:
    1. Use the shell GDB command to run the ps command and find the program’s process ID (pid): 
      (gdb) shell ps -C program -o pid h
       
       pid

      Replace program with a file name or path to the program.

    2. Use the attach command to attach GDB to the program: 
      (gdb) attach pid

      Replace pid by an actual process ID number from the ps output.

      Note

      In some cases, GDB might not be able to find the corresponding executable file. Use the file command to specify the path:

      (gdb) file path/to/program

Program execution control with GDB
Copy link

When the GNU Debugger (GDB) has been attached to a program, you can use several commands to control the execution of the program. These commands parse code, set breakpoints, and control program flow during debugging sessions.

To use these GDB commands effectively, you must have debugging information available through these methods:

  • The program is compiled and built with debugging information
  • The relevant debuginfo packages are installed
  • GDB must be attached to the program to be debugged

GDB provides commands for stepping through and controlling program execution:

r (run): Start the execution of the program. If run is executed with any arguments, those arguments are passed on to the executable as if the program has been started normally. Users normally issue this command after setting breakpoints.

start: Start the execution of the program but stop at the beginning of the program’s main function. If start is executed with any arguments, those arguments are passed on to the executable as if the program has been started normally.

c (continue): Continue the execution of the program from the current state. The execution of the program will continue until one of the following becomes true:

  • A breakpoint is reached.
  • A specified condition is satisfied.
  • A signal is received by the program.
  • An error occurs.
  • The program terminates.

n (next): Continue the execution of the program from the current state, until the next line of code in the current source file is reached. The execution of the program will continue until one of the following becomes true:

  • A breakpoint is reached.
  • A specified condition is satisfied.
  • A signal is received by the program.
  • An error occurs.
  • The program terminates.

s (step): The step command also halts execution at each sequential line of code in the current source file. However, if the execution is currently stopped at a source line containing a function call, GDB stops the execution after entering the function call (rather than executing it).

untillocation: Continue the execution until the code location specified by the location option is reached.

fini (finish): Resume the execution of the program and halt when execution returns from a function. The execution of the program will continue until one of the following becomes true:

  • A breakpoint is reached.
  • A specified condition is satisfied.
  • A signal is received by the program.
  • An error occurs.
  • The program terminates.

q (quit): Terminate the execution and exit GDB.

Show program internal values with GDB
Copy link

Displaying the values of a program’s internal variables is important for understanding of what the program is doing. The GNU Debugger (GDB) offers multiple commands that you can use to inspect the internal variables. The following list describes the most useful of these commands.

p (print)

Display the value of the argument given. Usually, the argument is the name of a variable of any complexity, from a simple single value to a structure. An argument can also be an expression valid in the current language, including the use of program variables and library functions, or functions defined in the program being tested.

It is possible to extend GDB with pretty-printer Python or Guile scripts for customized display of data structures (such as classes, structs) using the print command.

bt (backtrace)

Display the chain of function calls used to reach the current execution point, or the chain of functions used up until execution was terminated. This is useful for investigating serious bugs (such as segmentation faults) with elusive causes.

Adding the full option to the backtrace command displays local variables, too.

It is possible to extend GDB with frame filter Python scripts for customized display of data displayed using the bt and info frame commands. The term frame refers to the data associated with a single function call.

info

The info command is a generic command to provide information about various items. It takes an option specifying the item to describe.

  • The info args command displays options of the function call that is the currently selected frame.
  • The info locals command displays local variables in the currently selected frame.

For a list of the possible items, run the command help info in a GDB session:


              (gdb) help info
l (list)
Show program source code. When the program has been started, but is currently stopped, this command lists the source code where the program is currently stopped, along with a few lines of context. Before the program is started, this will list the main function. While not strictly a command to show internal state, list helps the user understand what changes to the internal state will happen in the next step of the program’s execution.

Use GDB breakpoints to stop execution at defined code locations
Copy link

During a debugging session, you often need to investigate only specific sections of code. Breakpoints are markers that instruct GDB to stop execution at a defined location. Because each breakpoint maps to a source line, you must specify the correct source file and line number when placing them.

Procedure
Copy link

  • To place a breakpoint.
    1. Specify the name of the source code file and the line in that file: 
      
                (gdb) br file:line
    2. When file is not present, name of the source file at the current point of execution is used: 
      
                (gdb) br line
    3. Alternatively, use a function name to put the breakpoint on its start: 
      
                (gdb) br function_name
    4. A program might encounter an error after a certain number of iterations of a task. To specify an additional condition to halt execution: 
      
                (gdb) br file:line if condition

      Replace condition with a condition in the C or C++ language. The placeholders file and line match the syntax shown in the first breakpoint step in this procedure.

  • To inspect the status of all breakpoints and watchpoints: 
    
            (gdb) info br
  • To remove a breakpoint by using its number as displayed in the output of info br: 
    
            (gdb) delete number
  • To remove a breakpoint at a given location: 
    
            (gdb) clear file:line

Use GDB watchpoints to stop execution on data access and changes
Copy link

GDB watchpoints pause execution when data changes or is accessed. Use them to debug unexpected variable corruption when the cause is unknown. Setting a watchpoint stops the program at the exact moment of access to inspect the state and find the root cause.

Procedure
Copy link

  • To place a watchpoint for data change (write): 
    
            (gdb) watch expression

    Replace expression with an expression that describes what you want to watch. For variables, expression is equal to the name of the variable.

  • To place a watchpoint for data access (read): 
    
            (gdb) rwatch expression
  • To place a watchpoint for any data access (both read and write): 
    
            (gdb) awatch expression
  • To inspect the status of all watchpoints and breakpoints: 
    
            (gdb) info br
  • To remove a watchpoint: 
    
            (gdb) delete num

    Replace the num option with the number reported by the info br command.

Debug forking or threaded programs with GDB
Copy link

Some programs use forking or threads to achieve parallel code execution. To debug multiple simultaneous execution paths, you can use a variety of commands based on your use case.

Debugging forked programs with GDB

Forking is a situation when a program (parent) creates an independent copy of itself (child). Use the following settings and commands to affect what GDB does when a fork occurs:

  • The follow-fork-mode setting controls whether GDB follows the parent or the child after the fork.
    set follow-fork-mode parent
    After a fork, debug the parent process. This is the default.
    set follow-fork-mode child
    After a fork, debug the child process.
    show follow-fork-mode
    Display the current setting of follow-fork-mode.
  • The set detach-on-fork setting controls whether the GDB keeps control of the other (not followed) process or leaves it to run.
    set detach-on-fork on
    The process which is not followed (depending on the value of follow-fork-mode) is detached and runs independently. This is the default.
    set detach-on-fork off
    GDB keeps control of both processes. The process which is followed (depending on the value of follow-fork-mode) is debugged as usual, while the other is suspended.
    show detach-on-fork
    Display the current setting of detach-on-fork.
Debugging Threaded Programs with GDB
GDB has the ability to debug individual threads, and to manipulate and examine them independently. To make GDB stop only the thread that is examined, use the commands set non-stop on and set target-async on. You can add these commands to the .gdbinit file. After that functionality is turned on, GDB is ready to conduct thread debugging.

GDB uses a concept of current thread. By default, commands apply to the current thread only.

info threads
Display a list of threads with their id and gid numbers, indicating the current thread.
thread id
Set the thread with the specified id as the current thread.
thread apply idscommand
Apply the command command to all threads listed by ids. The ids option is a space-separated list of thread ids. A special value all applies the command to all threads.
break location thread id if condition
Set a breakpoint at a certain location with a certain condition only for the thread number id.
watch expression thread id
Set a watchpoint defined by expression only for the thread number id.
command&
Run command command and return immediately to the gdb prompt (gdb), continuing any code execution in the background.
interrupt
Halt execution in the background.