Chapter 19. Viewing and Managing Log Files

Log files are files that contain messages about the system, including the kernel, services, and applications running on it. There are different log files for different information. For example, there is a default system log file, a log file just for security messages, and a log file for cron tasks.

Log files can be very useful when trying to troubleshoot a problem with the system such as trying to load a kernel driver or when looking for unauthorized login attempts to the system. This chapter discusses where to find log files, how to view log files, and what to look for in log files.

Some log files are controlled by a daemon called rsyslogd. A list of log files maintained by rsyslogd can be found in the /etc/rsyslog.conf configuration file.

rsyslog is an enhanced, multi-threaded syslog daemon which replaced the sysklogd daemon. rsyslog supports the same functionality as sysklogd and extends it with enhanced filtering, encryption protected relaying of messages, various configuration options, or support for transportation via the TCP or UDP protocols. Note that rsyslog is compatible with sysklogd.

19.1. Configuring rsyslog

The main configuration file for rsyslog is /etc/rsyslog.conf. It consists of global directives, rules or comments (any empty lines or any text following a hash sign (#)). Both, global directives and rules are extensively described in the sections below.

19.1.1. Global Directives

Global directives specify configuration options that apply to the rsyslogd daemon. They usually specify a value for a specific pre-defined variable that affects the behavior of the rsyslogd daemon or a rule that follows. All of the global directives must start with a dollar sign ($). Only one directive can be specified per line. The following is an example of a global directive that specifies the maximum size of the syslog message queue:

$MainMsgQueueSize 50000

The default size defined for this directive (10,000 messages) can be overridden by specifying a different value (as shown in the example above).

You may define multiple directives in your /etc/rsyslog.conf configuration file. A directive affects the behavior of all configuration options until another occurrence of that same directive is detected.

A comprehensive list of all available configuration directives and their detailed description can be found in /usr/share/doc/rsyslog-<version-number>/rsyslog_conf_global.html.

19.1.2. Modules

Due to its modular design, rsyslog offers a variety of modules which provide dynamic functionality. Note that modules can be written by third parties. Most modules provide additional inputs (see Input Modules below) or outputs (see Output Modules below). Other modules provide special functionality specific to each module. The modules may provide additional configuration directives that become available after a module is loaded. To load a module, use the following syntax:

$ModLoad <MODULE>

where $ModLoad is the global directive that loads the specified module and <MODULE> represents your desired module. For example, if you want to load the Text File Input Module (imfile — enables rsyslog to convert any standard text files into syslog messages), specify the following line in your /etc/rsyslog.conf configuration file:

$ModLoad imfile

rsyslog offers a number of modules which are split into these main categories:

Input Modules — Input modules gather messages from various sources. The name of an input module always starts with the im prefix, such as imfile, imrelp, etc.
Output Modules — Output modules provide a facility to store messages into various targets such as sending them across network, storing them in a database or encrypting them. The name of an output module always starts with the om prefix, such as omsnmp, omrelp, etc.
Filter Modules — Filter modules provide the ability to filter messages according to specified rules. The name of a filter module always starts with the fm prefix.
Parser Modules — Parser modules use the message parsers to parse message content of any received messages. The name of a parser module always starts with the pm prefix, such as pmrfc5424, pmrfc3164, etc.
Message Modification Modules — Message modification modules change the content of a syslog message. The message modification modules only differ in their implementation from the output and filter modules but share the same interface.
String Generator Modules — String generator modules generate strings based on the message content and strongly cooperate with the template feature provided by rsyslog. For more information on templates, refer to Section 19.1.3.3, “Templates”. The name of a string generator module always starts with the sm prefix, such as smfile, smtradfile, etc.
Library Modules — Library modules generally provide functionality for other loadable modules. These modules are loaded automatically by rsyslog when needed and cannot be configured by the user.

A comprehensive list of all available modules and their detailed description can be found at http://www.rsyslog.com/doc/rsyslog_conf_modules.html

Make sure you use trustworthy modules only

Note that when rsyslog loads any modules, it provides them with access to some of its functions and data. This poses a possible security threat. To minimize security risks, use trustworthy modules only.

19.1.3. Rules

A rule is specified by a filter part, which selects a subset of syslog messages, and an action part, which specifies what to do with the selected messages. To define a rule in your /etc/rsyslog.conf configuration file, define both, a filter and an action, on one line and separate them with one or more spaces or tabs. For more information on filters, refer to Section 19.1.3.1, “Filter Conditions” and for information on actions, refer to Section 19.1.3.2, “Actions”.

19.1.3.1. Filter Conditions

rsyslog offers various ways how to filter syslog messages according to various properties. This sections sums up the most used filter conditions.

Facility/Priority-based filters

The most used and well-known way to filter syslog messages is to use the facility/priority-based filters which filter syslog messages based on two conditions: facility and priority. To create a selector, use the following syntax:

<FACILITY>.<PRIORITY>

where:

<FACILITY> specifies the subsystem that produces a specific syslog message. For example, the mail subsystem handles all mail related syslog messages. <FACILITY> can be represented by one of these keywords: auth, authpriv, cron, daemon, kern, lpr, mail, news, syslog, user, uucp, and local0 through local7.
<PRIORITY> specifies a priority of a syslog message. <PRIORITY> can be represented by one of these keywords (listed in an ascending order): debug, info, notice, warning, err, crit, alert, and emerg.
By preceding any priority with an equal sign (=), you specify that only syslog messages with that priority will be selected. All other priorities will be ignored. Conversely, preceding a priority with an exclamation mark (!) selects all syslog messages but those with the defined priority. By not using either of these two extensions, you specify a selection of syslog messages with the defined or higher priority.

In addition to the keywords specified above, you may also use an asterisk (*) to define all facilities or priorities (depending on where you place the asterisk, before or after the dot). Specifying the keyword none serves for facilities with no given priorities.

To define multiple facilities and priorities, simply separate them with a comma (,). To define multiple filters on one line, separate them with a semi-colon (;).

The following are a few examples of simple facility/priority-based filters:

kern.*    # Selects all kernel syslog messages with any priority

mail.crit    # Selects all mail syslog messages with priority crit and higher.

cron.!info,!debug    # Selects all cron syslog messages except those with the info or debug priority.

Property-based filters

Property-based filters let you filter syslog messages by any property, such as timegenerated or syslogtag. For more information on properties, refer to Section 19.1.3.3.2, “Properties”. Each of the properties specified in the filters lets you compare it to a specific value using one of the compare-operations listed in Table 19.1, “Property-based compare-operations”.

Table 19.1. Property-based compare-operations

Compare-operation	Description
`contains`	Checks whether the provided string matches any part of the text provided by the property.
`isequal`	Compares the provided string against all of the text provided by the property.
`startswith`	Checks whether the provided string matches a prefix of the text provided by the property.
`regex`	Compares the provided POSIX BRE (Basic Regular Expression) regular expression against the text provided by the property.
`ereregex`	Compares the provided POSIX ERE (Extended Regular Expression) regular expression against the text provided by the property.

To define a property-based filter, use the following syntax:

:<PROPERTY>, [!]<COMPARE_OPERATION>, "<STRING>"

where:

The <PROPERTY> attribute specifies the desired property (for example, timegenerated, hostname, etc.).
The optional exclamation point (!) negates the output of the compare-operation (if prefixing the compare-operation).
The <COMPARE_OPERATION> attribute specifies one of the compare-operations listed in Table 19.1, “Property-based compare-operations”.
The <STRING> attribute specifies the value that the text provided by the property is compared to. To escape certain character (for example a quotation mark (")), use the backslash character (\).

The following are few examples of property-based filters:

The following filter selects syslog messages which contain the string error in their message text:
```
:msg, contains, "error"
```
The following filter selects syslog messages received from the hostname host1:
```
:hostname, isequal, "host1"
```
The following filter selects syslog messages which do not contain any mention of the words fatal and error with any or no text between them (for example, fatal lib error):
```
:msg, !regex, "fatal .* error"
```

Expression-based filters

Expression-based filters select syslog messages according to defined arithmetic, boolean or string operations. Expression-based filters use rsyslog's own scripting language. The syntax of this language is defined in /usr/share/doc/rsyslog-<version-number>/rscript_abnf.html along with examples of various expression-based filters.

To define an expression-based filter, use the following syntax:

if <EXPRESSION> then <ACTION>

where:

The <EXPRESSION> attribute represents an expression to be evaluated, for example: $msg startswith 'DEVNAME' or $syslogfacility-text == 'local0'.
The <ACTION> attribute represents an action to be performed if the expression returns the value true.

Define an expression-based filter on a single line

When defining an expression-based filter, it must be defined on a single line.

Do not use regular expressions

Regular expressions are currently not supported in expression-based filters.

BSD-style blocks

rsyslog supports BSD-style blocks inside the /etc/rsyslog.conf configuration file. Each block consists of rules which are preceded with a program or hostname label. Use the '!<PROGRAM>' or '-<PROGRAM>' labels to include or exclude programs, respectively. Use the '+<HOSTNAME> ' or '-<HOSTNAME> ' labels include or exclude hostnames, respectively.

Example 19.1, “BSD-style block” shows a BSD-style block that saves all messages generated by yum to a file.

Example 19.1. BSD-style block

!yum
*.*      /var/log/named.log

19.1.3.2. Actions

Actions specify what is to be done with the messages filtered out by an already-defined selector. The following are some of the actions you can define in your rule:

Saving syslog messages to log files

The majority of actions specify to which log file a syslog message is saved. This is done by specifying a file path after your already-defined selector. The following is a rule comprised of a selector that selects all cron syslog messages and an action that saves them into the /var/log/cron.log log file:

cron.* /var/log/cron.log

Use a dash mark (-) as a prefix of the file path you specified if you want to omit syncing the desired log file after every syslog message is generated.

Your specified file path can be either static or dynamic. Static files are represented by a simple file path as was shown in the example above. Dynamic files are represented by a template and a question mark (?) prefix. For more information on templates, refer to Section 19.1.3.3.1, “Generating dynamic file names”.

If the file you specified is an existing tty or /dev/console device, syslog messages are sent to standard output (using special tty-handling) or your console (using special /dev/console-handling) when using the X Window System, respectively.

Sending syslog messages over the network

rsyslog allows you to send and receive syslog messages over the network. This feature allows to administer syslog messages of multiple hosts on one machine. To forward syslog messages to a remote machine, use the following syntax:

@[(<OPTION>)]<HOST>:[<PORT>]

where:

The at sign (@) indicates that the syslog messages are forwarded to a host using the UDP protocol. To use the TCP protocol, use two at signs with no space between them (@@).
The <OPTION> attribute can be replaced with an option such as z<NUMBER>. This option enables zlib compression for syslog messages; the <NUMBER> attribute specifies the level of compression. To define multiple options, simply separate each one of them with a comma (,).
The <HOST> attribute specifies the host which receives the selected syslog messages.
The <PORT> attribute specifies the host machine's port.

When specifying an IPv6 address as the host, enclose the address in square brackets ([, ]).

The following are some examples of actions that forward syslog messages over the network (note that all actions are preceded with a selector that selects all messages with any priority):

*.* @192.168.0.1    # Forwards messages to 192.168.0.1 via the UDP protocol

*.* @@example.com:18    # Forwards messages to "example.com" using port 18 and the TCP protocol

*.* @(z9)[2001::1]    # Compresses messages with zlib (level 9 compression)
                      # and forwards them to 2001::1 using the UDP protocol

Output channels

Output channels are primarily used for log file rotation (for more info on log file rotation, refer to Section 19.2.1, “Configuring logrotate”), that is, to specify the maximum size a log file can grow to. To define an output channel, use the following syntax:

$outchannel <NAME>, <FILE_NAME>, <MAX_SIZE>, <ACTION>

where:

The <NAME> attribute specifies the name of the output channel.
The <FILE_NAME> attribute specifies the name of the output file.
The <MAX_SIZE> attribute represents the maximum size the specified file (in <FILE_NAME>) can grow to. This value is specified in bytes.
The <ACTION> attribute specifies the action that is taken when the maximum size, defined in <MAX_SIZE>, is hit.

Example 19.2, “Output channel log rotation” shows a simple log rotation through the use of an output channel. First, the output channel is defined via the $outchannel directive and then used in a rule which selects every syslog message with any priority and executes the previously-defined output channel on the acquired syslog messages. Once the limit (in the example 100 MB) is hit, the /home/joe/log_rotation_script is executed. This script can contain anything from moving the file into a different folder, editing specific content out of it, or simply removing it.

Example 19.2. Output channel log rotation

$outchannel log_rotation,/var/log/test_log.log, 104857600, /home/joe/log_rotation_script

*.* $log_rotation

Support for output channels is to be removed in the future

Output channels are currently supported by rsyslog, however, they are planned to be removed in the nearby future.

Sending syslog messages to specific users

rsyslog can send syslog messages to specific users by simply specifying a username of the user you wish to send the messages to. To specify more than one user, separate each username with a comma (,). To send messages to every user that is currently logged on, use an asterisk (*).

Executing a program

rsyslog lets you execute a program for selected syslog messages and uses the system() call to execute the program in shell. To specify a program to be executed, prefix it with a caret character (^). Consequently, specify a template that formats the received message and passes it to the specified executable as a one line parameter (for more information on templates, refer to Section 19.1.3.3, “Templates”). In the following example, any syslog message with any priority is selected, formatted with the template template and passed as a parameter to the test-program program, which is then executed with the provided parameter:

*.* ^test-program;template

Be careful when using the shell execute action

When accepting messages from any host, and using the shell execute action, you may be vulnerable to command injection. An attacker may try to inject and execute commands specified by the attacker in the program you specified (in your action) to be executed. To avoid any possible security threats, thoroughly consider the use of the shell execute action.

Inputting syslog messages in a database

Selected syslog messages can be directly written into a database table using the database writer action. The database writer uses the following syntax:

:<PLUGIN>:<DB_HOST>,<DB_NAME>,<DB_USER>,<DB_PASSWORD>;[<TEMPLATE>]

where:

The <PLUGIN> calls the specified plug-in that handles the database writing (for example, the ommysql plug-in).
The <DB_HOST> attribute specifies the database hostname.
The <DB_NAME> attribute specifies the name of the database.
The <DB_USER> attribute specifies the database user.
The <DB_PASSWORD> attribute specifies the password used with the aforementioned database user.
The <TEMPLATE> attribute specifies an optional use of a template that modifies the syslog message. For more information on templates, refer to Section 19.1.3.3, “Templates”.

Using MySQL and PostgreSQL

Currently, rsyslog provides support for MySQL (for more information, refer to /usr/share/doc/rsyslog-<version-number>/rsyslog_mysql.html) and PostgreSQL databases only. In order to use the MySQL and PostgreSQL database writer functionality, install the rsyslog-mysql and rsyslog-pgsql packages installed, respectively. Also, make sure you load the appropriate modules in your /etc/rsyslog.conf configuration file:

$ModLoad ommysql    # Output module for MySQL support
$ModLoad ompgsql    # Output module for PostgreSQL support

For more information on rsyslog modules, refer to Section 19.1.2, “Modules”.

Alternatively, you may use a generic database interface provided by the omlibdb module. However, this module is currently not compiled.

Discarding syslog messages

To discard your selected messages, use the tilde character (~). The following rule discards any cron syslog messages:

cron.* ~

For each selector, you are allowed to specify multiple actions. To specify multiple actions for one selector, write each action on a separate line and precede it with an ampersand character (&). Only the first action is allowed to have a selector specified on its line. The following is an example of a rule with multiple actions:

kern.=crit joe
& ^test-program;temp
& @192.168.0.1

In the example above, all kernel syslog messages with the critical priority (crit) are send to user joe, processed by the template temp and passed on to the test-program executable, and forwarded to 192.168.0.1 via the UDP protocol.

Specifying multiple actions improves the overall performance of the desired outcome since the specified selector has to be evaluated only once.

Note that any action can be followed by a template that formats the message. To specify a template, suffix an action with a semicolon (;) and specify the name of the template.

Using templates

A template must be defined before it is used in an action, otherwise, it is ignored.

For more information on templates, refer to Section 19.1.3.3, “Templates”.

19.1.3.3. Templates

Any output that is generated by rsyslog can be modified and formatted according to your needs through the use of templates. To create a template use the following syntax:

$template <TEMPLATE_NAME>,"text %<PROPERTY>% more text", [<OPTION>]

where:

$template is the template directive that indicates that the text following it, defines a template.
<TEMPLATE_NAME> is the name of the template. Use this name to refer to the template.
Anything between the two quotation marks ("…") is the actual template text. Within this text, you are allowed to escape characters in order to use their functionality, such as \n for new line or \r for carriage return. Other characters, such as % or ", have to be escaped in case you want to those characters literally.
The text specified within two percent signs (%) specifies a property that is consequently replaced with the property's actual value. For more information on properties, refer to Section 19.1.3.3.2, “Properties”
The <OPTION> attribute specifies any options that modify the template functionality. Do not mistake these for property options, which are defined inside the template text (between "…"). The currently supported template options are sql and stdsql used for formatting the text as an SQL query.
The sql and stdsql options
Note that the database writer (for more information, refer to section Inputting syslog messages in a database in Section 19.1.3.2, “Actions”) checks whether the sql and stdsql options are specified in the template. If they are not, the database writer does not perform any action. This is to prevent any possible security threats, such as SQL injection.

19.1.3.3.1. Generating dynamic file names

Templates can be used to generate dynamic file names. By specifying a property as a part of the file path, a new file will be created for each unique property. For example, use the timegenerated property to generate a unique file name for each syslog message:

$template DynamicFile,"/var/log/test_logs/%timegenerated%-test.log"

Keep in mind that the $template directive only specifies the template. You must use it inside a rule for it to take effect:

*.* ?DynamicFile

19.1.3.3.2. Properties

Properties defined inside a template (within two percent signs (%)) allow you to access various contents of a syslog message through the use of a property replacer. To define a property inside a template (between the two quotation marks ("…")), use the following syntax:

%<PROPERTY_NAME>[:<FROM_CHAR>:<TO_CHAR>:<OPTION>]%

where:

The <PROPERTY_NAME> attribute specifies the name of a property. A comprehensible list of all available properties and their detailed description can be found in /usr/share/doc/rsyslog-<version-number>/property_replacer.html under the section Available Properties.
<FROM_CHAR> and <TO_CHAR> attributes denote a range of characters that the specified property will act upon. Alternatively, regular expressions can be used to specify a range of characters. To do so, specify the letter R as the <FROM_CHAR> attribute and specify your desired regular expression as the <TO_CHAR> attribute.
The <OPTION> attribute specifies any property options. A comprehensible list of all available properties and their detailed description can be found in /usr/share/doc/rsyslog-<version-number>/property_replacer.html under the section Property Options.

The following are some examples of simple properties:

The following property simply obtains the whole message text of a syslog message:
```
%msg%
```
The following property obtains the first two characters of the message text of a syslog message:
```
%msg:1:2%
```
The following property obtains the whole message text of a syslog message and drops its last line feed character:
```
%msg:::drop-last-lf%
```
The following property obtains the first 10 characters of the timestamp that is generated when the syslog message is received and formats it according to the RFC 3999 date standard.
```
%timegenerated:1:10:date-rfc3339%
```

19.1.3.3.3. Template Examples

This section presents few examples of rsyslog templates.

Example 19.3, “A verbose syslog message template” shows a template that formats a syslog message so that it outputs the message's severity, facility, the timestamp of when the message was received, the hostname, the message tag, the message text, and ends with a new line.

Example 19.3. A verbose syslog message template

$template verbose,"%syslogseverity%,%syslogfacility%,%timegenerated%,%HOSTNAME%,%syslogtag%,%msg%\n"

Example 19.4, “A wall message template” shows a template that resembles a traditional wall message (a message that is send to every user that is logged in and has their mesg(1) permission set to yes). This template outputs the message text, along with a hostname, message tag and a timestamp, on a new line (using \r and \n) and rings the bell (using \7).

Example 19.4. A wall message template

$template wallmsg,"\r\n\7Message from syslogd@%HOSTNAME% at %timegenerated% ...\r\n %syslogtag% %msg%\n\r"

Example 19.5, “A database formatted message template” shows a template that formats a syslog message so that it can be used as a database query. Notice the use of the sql option at the end of the template specified as the template option. It tells the database writer to format the message as an MySQL SQL query.

Example 19.5. A database formatted message template

$template dbFormat,"insert into SystemEvents (Message, Facility,FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%',%syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",sql

rsyslog also contains a set of predefined templates identified by the RSYSLOG_ prefix. It is advisable to not create a template using this prefix to avoid any conflicts. The following list shows these predefined templates along with their definitions.

RSYSLOG_DebugFormat

"Debug line with all properties:\nFROMHOST: '%FROMHOST%', fromhost-ip: '%fromhost-ip%', HOSTNAME: '%HOSTNAME%', PRI: %PRI%,\nsyslogtag '%syslogtag%', programname: '%programname%', APP-NAME: '%APP-NAME%', PROCID: '%PROCID%', MSGID: '%MSGID%',\nTIMESTAMP: '%TIMESTAMP%', STRUCTURED-DATA: '%STRUCTURED-DATA%',\nmsg: '%msg%'\nescaped msg: '%msg:::drop-cc%'\nrawmsg: '%rawmsg%'\n\n\"

RSYSLOG_SyslogProtocol23Format

"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n\"

RSYSLOG_FileFormat

"%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\"

RSYSLOG_TraditionalFileFormat

"%TIMESTAMP% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n\"

RSYSLOG_ForwardFormat

"<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%\"

RSYSLOG_TraditionalForwardFormat

"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag:1:32%%msg:::sp-if-no-1st-sp%%msg%\"

19.1.4. rsyslog Command Line Configuration

Some of rsyslog's functionality can be configured through the command line options, as sysklogd's can. Note that as of version 3 of rsyslog, this method was deprecated. To enable some of these option, you must specify the compatibility mode rsyslog should run in. However, configuring rsyslog through the command line options should be avoided.

To specify the compatibility mode rsyslog should run in, use the -c option. When no parameter is specified, rsyslog tries to be compatible with sysklogd. This is partially achieved by activating configuration directives that modify your configuration accordingly. Therefore, it is advisable to supply this option with a number that matches the major version of rsyslog that is in use and update your /etc/rsyslog.conf configuration file accordingly. If you want to, for example, use sysklogd options (which were deprecated in version 3 of rsyslog), you can specify so by executing the following command:

~]# rsyslogd -c 2

Options that are passed to the rsyslogd daemon, including the backward compatibility mode, can be specified in the /etc/sysconfig/rsyslog configuration file.

For more information on various rsyslogd options, refer to man rsyslogd.