Record application interactions

The executable code of applications interacts with the operating system and shared libraries. Recording an activity log of these interactions can provide enough insight into application behavior without debugging the actual application code, or help pinpoint conditions where a bug manifests.

Tools to record application interactions
Copy link

To record application interactions, you can use several tools are available in RHEL. For system calls use strace, for library calls use ltrace, and for advanced probing SystemTap. Select the appropriate tool to log specific runtime behaviors and diagnose integration issues.

strace

The strace tool primarily enables logging of system calls (kernel functions) used by an application.

  • The strace tool can provide a detailed output about calls, because strace interprets parameters and results with knowledge of the underlying kernel code. Numbers are turned into the corresponding constant names, bitwise combined flags expanded to flag list, pointers to character arrays dereferenced to provide the actual string, and more. Support for more recent kernel features might be lacking.
  • To reduce the amount of captured data, filter the traced calls.
  • The use of strace does not require any particular configuration except for configuring the log filter.
  • Tracing the application code with strace results in significant slowdown of the application’s execution. As a result, strace is not suitable for many production deployments. As an alternative, consider using ltrace or SystemTap.
  • The version of strace available in Red Hat Developer Toolset can also perform system call tampering. This capability is useful for debugging.
ltrace

The ltrace tool enables logging of an application’s user space calls into shared objects (dynamic libraries).

  • The ltrace tool enables tracing calls to any library.
  • To reduce the amount of captured data, filter the traced calls.
  • The use of ltrace does not require any particular configuration except for configuring the log filter.
  • The ltrace tool is lightweight and fast, offering an alternative to strace: it is possible to trace the corresponding interfaces in libraries such as glibc with ltrace instead of tracing kernel functions with strace.
  • Because ltrace does not handle a known set of calls such as strace, it does not attempt to explain the values passed to library functions. The ltrace output contains only raw numbers and pointers. The interpretation of ltrace output requires consulting the actual interface declarations of the libraries present in the output.
Note

In Red Hat Enterprise Linux 10, a known issue prevents ltrace from tracing system executable files. This limitation does not apply to executable files built by users.
SystemTap

SystemTap is an instrumentation platform for probing running processes and kernel activity on the Linux system. SystemTap uses its own scripting language for programming custom event handlers.

  • Compared to using strace and ltrace, scripting the logging means more work in the initial setup phase. However, the scripting capabilities extend SystemTap’s usefulness beyond just producing logs.
  • SystemTap works by creating and inserting a kernel module. The use of SystemTap is efficient and does not create a significant slowdown of the system or application execution on its own.
  • SystemTap includes a set of usage examples.
GDB

The GNU Debugger (GDB) is primarily meant for debugging, not logging. However, some of its features make it useful even in the scenario where an application’s interaction is the primary activity of interest.

  • With GDB, it is possible to conveniently combine the capture of an interaction event with immediate debugging of the subsequent execution path.
  • GDB is best suited for analyzing response to infrequent or singular events, after the initial identification of problematic situation by other tools. Using GDB in any scenario with frequent events becomes inefficient or even impossible.

Monitor system calls of an application with strace
Copy link

To monitor the system (kernel) calls performed by an application, use the strace tool.

Before you begin
Copy link

Procedure
Copy link

  1. Identify the system calls to monitor.
    • Start strace and attach it to the program.
      • If the program you want to monitor is not running, start strace and specify the program: 
        $ strace -fvttTyy -s 256 -e trace=callprogram
      • If the program is already running, find its process id (pid): 
        $ ps -C program
      • Attach strace to the process: 
        $ strace -fvttTyy -s 256 -e trace=call -ppid
    • Replace call with the system calls to be displayed. You can use the -e trace=call option multiple times. If left out, strace will display all system call types. See the strace(1) manual page for more information.
    • If you do not want to trace any forked processes or threads, omit the -f option.
  2. The strace tool displays the system calls made by the application and their details.

    In most cases, an application and its libraries make a large number of calls and strace output displays immediately, if no filter for system calls is set.

  3. The strace tool exits when the program exits.

    To terminate the monitoring before the traced program exits, press Ctrl+C.

    • If strace started the program, the program terminates together with strace.
    • If you attached strace to an already running program, the program terminates together with strace.
  4. Analyze the list of system calls done by the application.
    • Problems with resource access or availability are present in the log as calls returning errors.
    • Values passed to the system calls and patterns of call sequences provide insight into the causes of the application’s behaviour.
    • If the application crashes, the important information is probably at the end of log.
    • The output contains a large amount of unnecessary information. However, you can construct a more precise filter for the system calls of interest and repeat the procedure. Note

      It is advantageous to both see the output and save it to a file. Use the tee command to achieve this:

      $ strace ... |& tee your_log_file.log

Monitor application’s library function calls with ltrace
Copy link

To monitor an application’s calls to functions available in libraries (shared objects), use the ltrace tool.

Before you begin
Copy link

  • You must have ltrace installed on the system. For details, see Installing tools for debugging applications. .Procedure
    1. Identify the libraries and functions of interest, if possible.
    2. Start ltrace and attach it to the program. Note

      In Red Hat Enterprise Linux 10, a known issue prevents ltrace from tracing system executable files. This limitation does not apply to executable files built by users.

      • If the program you want to monitor is not running, start ltrace and specify program: 
        $ ltrace -f -l library -e functionprogram
      • If the program is already running, find its process id (pid): 
        $ ps -C program
      • Attach ltrace to the process: 
        $ ltrace -f -l library -e function -ppidprogram
      • Use the -e, -f and -l options to filter the output:
        • Supply the function names to be displayed as function. The -e function option can be used multiple times. If left out, ltrace displays calls to all functions.
        • Instead of specifying functions, you can specify whole libraries with the -l library option. This option behaves similarly to the -e function option.
        • If you do not want to trace any forked processes or threads, omit the -f option.

        See the ltrace(1)_ manual page for more information.

    3. ltrace displays the library calls made by the application.

      In most cases, an application makes a large number of calls and ltrace output displays immediately, if no filter is set.

    4. ltrace exits when the program exits.

      To terminate the monitoring before the traced program exits, press ctrl+C.

  • If ltrace started the program, the program terminates together with ltrace.
  • If you attached ltrace to an already running program, the program terminates together with ltrace.
    1. Stop ltrace by pressing Ctrl+C.
    2. Analyze the list of library calls done by the application.
  • If the application crashes, the important information is probably at the end of log.
  • The output contains a large amount of unnecessary information. However, you can construct a more precise filter and repeat the procedure. Note

    It is advantageous to both see the output and save it to a file. Use the tee command to achieve this:

    $ ltrace ... |& tee your_log_file.log

Monitor application’s system calls with SystemTap
Copy link

To register custom event handlers for kernel events, use the SystemTap tool. SystemTap is more efficient than the strace tool, but requires more setup. Installing SystemTap also installs the strace.stp script, which provides strace-like functionality when using SystemTap on your system.

Before you begin
Copy link

  • SystemTap and the corresponding kernel packages must be installed on the system. For details, see Installing tools for measuring application performance. .Procedure
    1. Find the process ID (pid) of the process you want to monitor: 
      $ ps -aux
    2. Run SystemTap with the strace.stp script: 
      # stap /usr/share/systemtap/examples/process/strace.stp -x pid

      The value of pid is the process id.

      The script is compiled to a kernel module, which is then loaded. This introduces a slight delay between entering the command and getting the output.

    3. When the process performs a system call, the call name and its parameters are printed on the command line.
    4. The script exits when the process terminates, or when you press Ctrl+C.

Use GDB to intercept application system calls
Copy link

To stop program execution when the program performs specific system calls, use GNU Debugger (GDB) catchpoints, then inspect the program state and system call parameters at those points.

Before you begin
Copy link

  • You must understand the usage of GDB breakpoints. For details, see Using GDB breakpoints to stop execution at defined code locations.
  • GDB must be attached to the program. For details, see Attaching GDB to a process. .Procedure
    1. Set the catchpoint: 
      (gdb) catch syscall syscall-name

      The command catch syscall sets a special type of breakpoint that halts execution when the program performs a system call.

      The syscall-name option specifies the name of the call. You can specify multiple catchpoints for various system calls. Leaving out the syscall-name option causes GDB to stop on any system call.

    2. Start execution of the program.
      • If the program has not started execution, start it: 
        (gdb) r
      • If the program execution is halted, resume it: 
        (gdb) c
    3. GDB halts execution after the program performs any specified system call.
    4. Use further GDB commands to examine the program state and advance execution, according to the particular situation.
    5. To exit the GDB debugging session: 
      (gdb) q

Use GDB to intercept handling of signals by applications
Copy link

To stop the execution of a program under specific circumstances, you can use the GNU Debugger (GDB). To stop the execution when the program receives a signal from the operating system, use a GDB catchpoint.

Before you begin
Copy link

  • You must understand the usage of GDB breakpoints. For details, see Using GDB breakpoints to stop execution at defined code locations.
  • GDB must be attached to the program. For details, see Attaching GDB to a process. .Procedure
    1. Set the catchpoint: 
      (gdb) catch signal signal-type

      The command catch signal sets a special type of a breakpoint that halts execution when a signal is received by the program. The signal-type option specifies the type of the signal. Use the special value 'all' to catch all signals.

    2. Let the program run.
      • If the program has not started execution, start it: 
        (gdb) r
      • If the program execution is halted, resume it: 
        (gdb) c
    3. GDB halts execution after the program receives any specified signal.
    4. Continue by debugging the program code handling the signal, or resume execution with the knowledge of the signal being received.
    5. Later, to exit the GDB debugging session: 
      (gdb) q