Chapter 25. Automatic Bug Reporting Tool (ABRT)

The ABRT graphical user interface provides an easy-to-use front end for working in a desktop environment. You can install the required package by running the following command as the root user:

~]# yum install abrt-desktop

Upon installation, the ABRT notification applet is configured to start automatically when your graphical desktop session starts. You can verify that the ABRT applet is running by issuing the following command in a terminal:

~]$ ps -el | grep abrt-applet
0 S  500 2036 1824 0 80  0 - 61604 poll_s ?    00:00:00 abrt-applet

If the applet is not running, you can start it manually in your current desktop session by running the abrt-applet program:

~]$ abrt-applet &
[1] 2261

The command line interface is useful on headless machines, remote systems connected over a network, or in scripts. You can install the required package by running the following command as the root user:

~]# yum install abrt-cli

To receive email notifications about crashes detected by ABRT, you need to have the libreport-plugin-mailx package installed. You can install it by executing the following command as root:

~]# yum install libreport-plugin-mailx

By default, it sends notifications to the root user at the local machine. The email destination can be configured in the /etc/libreport/plugins/mailx.conf file.

To have notifications displayed in your console at login time, install the abrt-console-notification package as well.

ABRT can detect, analyze, and report various types of software failures. By default, ABRT is installed with support for the most common types of failures, such as crashes of C and C++ applications. Support for other types of failures is provided by independent packages. For example, to install support for detecting exceptions in applications written using the Java language, run the following command as root:

~]# yum install abrt-java-connector

See Section 25.4, “Detecting Software Problems” for a list of languages and software projects which ABRT supports. The section also includes a list of all corresponding packages that enable the detection of the various types of failures.

The abrtd daemon requires the abrt user to exist for file system operations in the /var/spool/abrt directory. When the abrt package is installed, it automatically creates the abrt user whose UID and GID is 173, if such user does not already exist. Otherwise, the abrt user can be created manually. In that case, any UID and GID can be chosen, because abrtd does not require a specific UID and GID.

The abrtd daemon is configured to start at boot time. You can use the following command to verify its current status:

~]$ systemctl is-active abrtd.service
active

If systemctl returns inactive or unknown, the daemon is not running. You can start it for the current session by entering the following command as root:

~]# systemctl start abrtd.service

You can use the same commands to start or check status of related error-detection services. For example, make sure the abrt-ccpp service is running if you want ABRT to detect C or C++ crashes. See Section 25.4, “Detecting Software Problems” for a list of all available ABRT detection services and their respective packages.

With the exception of the abrt-vmcore and abrt-pstoreoops services, which are only started when a kernel panic or kernel oops occurs, all ABRT services are automatically enabled and started at boot time when their respective packages are installed. You can disable or enable any ABRT service by using the systemctl utility as described in Chapter 10, Managing Services with systemd.

To test that ABRT works properly, use the kill command to send the SEGV signal to terminate a process. For example, start a sleep process and terminate it with the kill command in the following way:

~]$ sleep 100 &
[1] 2823
~]$ kill -s SIGSEGV 2823

ABRT detects a crash shortly after executing the kill command, and, provided a graphical session is running, the user is notified of the detected problem by the GUI notification applet. On the command line, you can check that the crash was detected by running the abrt-cli list command or by examining the crash dump created in the /var/spool/abrt/ directory. See Section 25.5, “Handling Detected Problems” for more information on how to work with detected crashes.

ABRT events use plugins to carry out the actual reporting operations. Plugins are compact utilities that the events call to process the content of problem-data directories. Using plugins, ABRT is capable of reporting problems to various destinations, and almost every reporting destination requires some configuration. For instance, Bugzilla requires a user name, password, and a URL pointing to an instance of the Bugzilla service.

Some configuration details can have default values (such as a Bugzilla URL), but others cannot have sensible defaults (for example, a user name). ABRT looks for these settings in configuration files, such as report_Bugzilla.conf, in the /etc/libreport/events/ or $HOME/.cache/abrt/events/ directories for system-wide or user-specific settings respectively. The configuration files contain pairs of directives and values.

These files are the bare minimum necessary for running events and processing the problem-data directories. The gnome-abrt and abrt-cli tools read the configuration data from these files and pass it to the events they run.

Additional information about events (such as their description, names, types of parameters that can be passed to them as environment variables, and other properties) is stored in event_name.xml files in the /usr/share/libreport/events/ directory. These files are used by both gnome-abrt and abrt-cli to make the user interface more friendly. Do not edit these files unless you want to modify the standard installation. If you intend to do that, copy the file to be modified to the /etc/libreport/events/ directory and modify the new file. These files can contain the following information:

For example, the report_Logger event accepts an output filename as a parameter. Using the respective event_name.xml file, the ABRT GUI determines which parameters can be specified for a selected event and allows the user to set the values for these parameters. The values are saved by the ABRT GUI and reused on subsequent invocations of these events. Note that the ABRT GUI saves configuration options using the GNOME Keyring tool and by passing them to events, it overrides data from text configuration files.

To open the graphical Configuration window, choose Automatic Bug Reporting Tool Preferences from within a running instance of the gnome-abrt application. This window shows a list of events that can be selected during the reporting process when using the GUI. When you select one of the configurable events, you can click the Configure button and modify the settings for that event.

Figure 25.1. Configuring ABRT Events

The following table shows a selection of the default analyzing, collecting, and reporting events provided by the standard installation of ABRT. The table lists each event’s name, identifier, configuration file from the /etc/libreport/events.d/ directory, and a brief description. Note that while the configuration files use the event identifiers, the ABRT GUI refers to the individual events using their names. Note also that not all of the events can be set up using the GUI. For information on how to define a custom event, see Section 25.3.2, “Creating Custom Events”.

Each event is defined by one rule structure in a respective configuration file. The configuration files are typically stored in the /etc/libreport/events.d/ directory. These configuration files are loaded by the main configuration file, /etc/libreport/report_event.conf. There is no need to edit the default configuration files because abrt will run the scripts contained in /etc/libreport/events.d/. This file accepts shell metacharacters (for example, *, $, ?) and interprets relative paths relatively to its location.

Each rule starts with a line with a non-space leading character, and all subsequent lines starting with the space character or the tab character are considered a part of this rule. Each rule consists of two parts, a condition part and a program part. The condition part contains conditions in one of the following forms:

where:

The program part consists of program names and shell-interpretable code. If all conditions in the condition part are valid, the program part is run in the shell. The following is an event example:

EVENT=post-create date > /tmp/dt
    echo $HOSTNAME uname -r

This event would overwrite the contents of the /tmp/dt file with the current date and time and print the host name of the machine and its kernel version on the standard output.

Here is an example of a more complex event, which is actually one of the predefined events. It saves relevant lines from the ~/.xsession-errors file to the problem report of any problem for which the abrt-ccpp service has been used, provided the crashed application had any X11 libraries loaded at the time of the crash:

EVENT=analyze_xsession_errors analyzer=CCpp dso_list~=./libX11.
    test -f ~/.xsession-errors || { echo "No ~/.xsession-errors"; exit 1; }
    test -r ~/.xsession-errors || { echo "Can't read ~/.xsession-errors"; exit 1; }
    executable=cat executable &&
    base_executable=${executable##*/} &&
    grep -F -e "$base_executable" ~/.xsession-errors | tail -999 >xsession_errors &&
    echo "Element 'xsession_errors' saved"

The set of possible events is not definitive. System administrators can add events according to their need in the /etc/libreport/events.d/ directory.

Currently, the following event names are provided with the standard ABRT and libreport installations:

ABRT can be configured to send initial anonymous reports, or μReports, of any detected issues or crashes automatically without any user interaction. When automatic reporting is turned on, the so called μReport, which is normally sent at the beginning of the crash-reporting process, is sent immediately after a crash is detected. This prevents duplicate support cases based on identical crashes. To enable the autoreporting feature, issue the following command as root:

~]# abrt-auto-reporting enabled

The above command sets the AutoreportingEnabled directive in the /etc/abrt/abrt.conf configuration file to yes. This system-wide setting applies to all users of the system. Note that by enabling this option, automatic reporting will also be enabled in the graphical desktop environment. To only enable autoreporting in the ABRT GUI, switch the Automatically send uReport option to YES in the Problem Reporting Configuration window. To open this window, choose Automatic Bug Reporting Tool ABRT Configuration from within a running instance of the gnome-abrt application. To launch the application, go to Applications Sundry Automatic Bug Reporting Tool .

Figure 25.2. Configuring ABRT Problem Reporting

Upon detection of a crash, by default, ABRT submits a μReport with basic information about the problem to Red Hat’s ABRT server. The server determines whether the problem is known and either provides a short description of the problem along with a URL of the reported case if known, or invites the user to report it if not known.

To change the default behavior of the autoreporting facility from sending a μReport, modify the value of the AutoreportingEvent directive in the /etc/abrt/abrt.conf configuration file to point to a different ABRT event. See Table 25.1, “Standard ABRT Events” for an overview of the standard events.

The abrt-ccpp service installs its own core-dump handler, which, when started, overrides the default value of the kernel’s core_pattern variable, so that C and C++ crashes are handled by abrtd. If you stop the abrt-ccpp service, the previously specified value of core_pattern is reinstated.

By default, the /proc/sys/kernel/core_pattern file contains the string core, which means that the kernel produces files with the core. prefix in the current directory of the crashed process. The abrt-ccpp service overwrites the core_pattern file to contain the following command:

|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e

This command instructs the kernel to pipe the core dump to the abrt-hook-ccpp program, which stores it in ABRT's dump location and notifies the abrtd daemon of the new crash. It also stores the following files from the /proc/PID/ directory (where PID is the ID of the crashed process) for debugging purposes: maps, limits, cgroup, status. See proc(5) for a description of the format and the meaning of these files.

The abrt-addon-python package installs a custom exception handler for Python applications. The Python interpreter then automatically imports the abrt.pth file installed in /usr/lib64/python2.7/site-packages/, which in turn imports abrt_exception_handler.py. This overrides Python’s default sys.excepthook with a custom handler, which forwards unhandled exceptions to abrtd via its Socket API.

To disable the automatic import of site-specific modules, and thus prevent the ABRT custom exception handler from being used when running a Python application, pass the -S option to the Python interpreter:

~]$ python -S file.py

In the above command, replace file.py with the name of the Python script you want to execute without the use of site-specific modules.

The rubygem-abrt package registers a custom handler using the at_exit feature, which is executed when a program ends. This allows for checking for possible unhandled exceptions. Every time an unhandled exception is captured, the ABRT handler prepares a bug report, which can be submitted to Red Hat Bugzilla using standard ABRT tools.

The ABRT Java Connector is a JVM agent that reports uncaught Java exceptions to abrtd. The agent registers several JVMTI event callbacks and has to be loaded into the JVM using the -agentlib command line parameter. Note that the processing of the registered callbacks negatively impacts the performance of the application. Use the following command to have ABRT catch exceptions from a Java class:

~]$ java -agentlib:abrt-java-connector=abrt=on $MyClass -platform.jvmtiSupported true

In the above command, replace $MyClass with the name of the Java class you want to test. By passing the abrt=on option to the connector, you ensure that the exceptions are handled by abrtd. In case you want to have the connector output the exceptions to standard output, omit this option.

The abrt-xorg service collects and processes information about crashes of the X.Org server from the /var/log/Xorg.0.log file. Note that no report is generated if a blacklisted X.org module is loaded. Instead, a not-reportable file is created in the problem-data directory with an appropriate explanation. You can find the list of offending modules in the /etc/abrt/plugins/xorg.conf file. Only proprietary graphics-driver modules are blacklisted by default.

By checking the output of kernel logs, ABRT is able to catch and process the so-called kernel oopses — non-fatal deviations from the correct behavior of the Linux kernel. This functionality is provided by the abrt-oops service.

ABRT can also detect and process kernel panics — fatal, non-recoverable errors that require a reboot, using the abrt-vmcore service. The service only starts when a vmcore file (a kernel-core dump) appears in the /var/crash/ directory. When a core-dump file is found, abrt-vmcore creates a new problem-data directory in the /var/spool/abrt/ directory and copies the core-dump file to the newly created problem-data directory. After the /var/crash/ directory is searched, the service is stopped.

For ABRT to be able to detect a kernel panic, the kdump service must be enabled on the system. The amount of memory that is reserved for the kdump kernel has to be set correctly. You can set it using the system-config-kdump graphical tool or by specifying the crashkernel parameter in the list of kernel options in the GRUB 2 menu. For details on how to enable and configure kdump, see the Red Hat Enterprise Linux 7 Kernel Crash Dump Guide. For information on making changes to the GRUB 2 menu see Chapter 26, Working with GRUB 2.

Using the abrt-pstoreoops service, ABRT is capable of collecting and processing information about kernel panics, which, on systems that support pstore, is stored in the automatically-mounted /sys/fs/pstore/ directory. The platform-dependent pstore interface (persistent storage) provides a mechanism for storing data across system reboots, thus allowing for preserving kernel panic information. The service starts automatically when kernel crash-dump files appear in the /sys/fs/pstore/ directory.

In the command line environment, the user is notified of new crashes on login, provided they have the abrt-console-notification package installed. The console notification looks like the following:

ABRT has detected 1 problem(s). For more info run: abrt-cli list --since 1398783164

To view detected problems, enter the abrt-cli list command:

~]$ abrt-cli list
id 6734c6f1a1ed169500a7bfc8bd62aabaf039f9aa
Directory:   /var/tmp/abrt/ccpp-2014-04-21-09:47:51-3430
count:     1
executable:   /usr/bin/sleep
package:    coreutils-8.22-11.el7
time:      Mon 21 Apr 2014 09:47:51 AM EDT
uid:      1000
Run 'abrt-cli report /var/tmp/abrt/ccpp-2014-04-21-09:47:51-3430' for creating a case in Red Hat Customer Portal

Each crash listed in the output of the abrt-cli list command has a unique identifier and a directory that can be used for further manipulation using abrt-cli.

To view information about just one particular problem, use the abrt-cli info command:

    abrt-cli info -d directory_or_id

To increase the amount of information displayed when using both the list and info sub-commands, pass them the -d (--detailed) option, which shows all stored information about the problems listed, including respective backtrace files if they have already been generated.

To analyze and report a certain problem, use the abrt-cli report command:

    abrt-cli report directory_or_id

Upon invocation of the above command, you will be asked to provide your credentials for opening a support case with Red Hat Customer Support. Next, abrt-cli opens a text editor with the content of the report. You can see what is being reported, and you can fill in instructions on how to reproduce the crash and other comments. You should also check the backtrace because the backtrace might be sent to a public server and viewed by anyone, depending on the problem-reporter event settings.

export VISUAL=emacs

When you are done with the report, save your changes and close the editor. If you have reported your problem to the Red Hat Customer Support database, a problem case is filed in the database. From now on, you will be informed about the problem-resolution progress via email you provided during the process of reporting. You can also monitor the problem case using the URL that is provided to you when the problem case is created or via emails received from Red Hat Support.

If you are certain that you do not want to report a particular problem, you can delete it. To delete a problem, so that ABRT does not keep information about it, use the command:

    abrt-cli rm directory_or_id

To display help about a particular abrt-cli command, use the --help option:

    abrt-cli command --help

The ABRT daemon broadcasts a D-Bus message whenever a problem report is created. If the ABRT applet is running in a graphical desktop environment, it catches this message and displays a notification dialog on the desktop. You can open the ABRT GUI using this dialog by clicking on the Report button. You can also open the ABRT GUI by selecting the Applications Sundry Automatic Bug Reporting Tool menu item.

Alternatively, you can run the ABRT GUI from the command line as follows:

~]$ gnome-abrt &

The ABRT GUI window displays a list of detected problems. Each problem entry consists of the name of the failing application, the reason why the application crashed, and the date of the last occurrence of the problem.

Figure 25.3. ABRT GUI

To access a detailed problem description, double-click on a problem-report line or click on the Report button while the respective problem line is selected. You can then follow the instructions to proceed with the process of describing the problem, determining how it should be analyzed, and where it should be reported. To discard a problem, click on the Delete button.