Chapter 23. Viewing and Managing Log Files

A rule is specified by a filter part, which selects a subset of syslog messages, and an action part, which specifies what to do with the selected messages. To define a rule in your /etc/rsyslog.conf configuration file, define both, a filter and an action, on one line and separate them with one or more spaces or tabs.

rsyslog offers various ways to filter syslog messages according to selected properties. The available filtering methods can be divided into Facility/Priority-based, Property-based, and Expression-based filters.

See the section called “Online Documentation” for more examples of various expression-based filters. RainerScript is the basis for rsyslog's new configuration format, see Section 23.3, “Using the New Configuration Format”

Actions specify what is to be done with the messages filtered out by an already defined selector. The following are some of the actions you can define in your rule:

Specifying Multiple Actions

For each selector, you are allowed to specify multiple actions. To specify multiple actions for one selector, write each action on a separate line and precede it with an ampersand (&) character:

FILTER ACTION
& ACTION
& ACTION

Specifying multiple actions improves the overall performance of the desired outcome since the specified selector has to be evaluated only once.

kern.=crit user1
& ^test-program;temp
& @192.168.0.1

Any action can be followed by a template that formats the message. To specify a template, suffix an action with a semicolon (;) and specify the name of the template. For more information on templates, see Section 23.2.3, “Templates”.

Any output that is generated by rsyslog can be modified and formatted according to your needs with the use of templates. To create a template use the following syntax in /etc/rsyslog.conf:

template(name=”TEMPLATE_NAME” type=”string” string="text %PROPERTY% more text" [option.OPTION="on"])

where:

Generating Dynamic File Names

Templates can be used to generate dynamic file names. By specifying a property as a part of the file path, a new file will be created for each unique property, which is a convenient way to classify syslog messages.

For example, use the timegenerated property, which extracts a time stamp from the message, to generate a unique file name for each syslog message:

  template(name=”DynamicFile” type=”list”) {
  constant(value=”/var/log/test_logs/”)
  property(name=”timegenerated”)
  constant(value”-test.log”)
}

Keep in mind that the $template directive only specifies the template. You must use it inside a rule for it to take effect. In /etc/rsyslog.conf, use the question mark (?) in an action definition to mark the dynamic file name template:

. ?DynamicFile

Properties

Properties defined inside a template (between two percent signs (%)) enable access various contents of a syslog message through the use of a property replacer. To define a property inside a template (between the two quotation marks (“…”)), use the following syntax:

%PROPERTY_NAME:FROM_CHAR:TO_CHAR:OPTION%

where:

The following are some examples of simple properties:

Template Examples

This section presents a few examples of rsyslog templates.

Example 23.8, “A verbose syslog message template” shows a template that formats a syslog message so that it outputs the message’s severity, facility, the time stamp of when the message was received, the host name, the message tag, the message text, and ends with a new line.

  template(name=”verbose” type=”list”) {
  property(name="syslogseverity”)
  property(name="syslogfacility”)
  property(name="timegenerated”)
  property(name="HOSTNAME”)
  property(name="syslogtag”)
  property(name="msg”)
  constant(value=”\n")
}

Example 23.9, “A wall message template” shows a template that resembles a traditional wall message (a message that is send to every user that is logged in and has their mesg(1) permission set to yes). This template outputs the message text, along with a host name, message tag and a time stamp, on a new line (using \r and \n) and rings the bell (using \7).

  template(name=”wallmsg” type=”list”) {
  constant(value="\r\n\7Message from syslogd@”)
  property(name="HOSTNAME”)
  constant(value=” at ")
  property(name="timegenerated”)
  constant(value=" ...\r\n ”)
  property(name="syslogtag”)
  constant(value=” “)
  property(name="msg”)
  constant(value=”\r\n”)
}

Example 23.10, “A database formatted message template” shows a template that formats a syslog message so that it can be used as a database query. Notice the use of the sql option at the end of the template specified as the template option. It tells the database writer to format the message as an MySQL SQL query.

template(name="dbFormat" type="list" option.sql="on") {
constant(value="insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag)")
constant(value=" values ('")
property(name="msg")
constant(value="', ")
property(name="syslogfacility")
constant(value=", '")
property(name="hostname")
constant(value="', ")
property(name="syslogpriority")
constant(value=", '")
property(name="timereported" dateFormat="mysql")
constant(value="', '")
property(name="timegenerated" dateFormat="mysql")
constant(value="', ")
property(name="iut")
constant(value=", '")
property(name="syslogtag")
constant(value="')")
}

rsyslog also contains a set of predefined templates identified by the RSYSLOG_ prefix. These are reserved for the syslog’s use and it is advisable to not create a template using this prefix to avoid conflicts. The following list shows these predefined templates along with their definitions.

Global directives are configuration options that apply to the rsyslogd daemon. They usually specify a value for a specific predefined variable that affects the behavior of the rsyslogd daemon or a rule that follows. All of the global directives are enclosed in a global configuration block. The following is an example of a global directive that specifies overriding local host name for log messages:

global(localHostname=”machineXY”)

You can define multiple directives in your /etc/rsyslog.conf configuration file. A directive affects the behavior of all configuration options until another occurrence of that same directive is detected. Global directives can be used to configure actions, queues and for debugging. A comprehensive list of all available configuration directives can be found in the section called “Online Documentation”. Currently, a new configuration format has been developed that replaces the $-based syntax (see Section 23.3, “Using the New Configuration Format”). However, classic global directives remain supported as a legacy format.

The following is a sample /etc/logrotate.conf configuration file:

# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# uncomment this if you want your log files compressed
compress

All of the lines in the sample configuration file define global options that apply to every log file. In our example, log files are rotated weekly, rotated log files are kept for four weeks, and all rotated log files are compressed by gzip into the .gz format. Any lines that begin with a hash sign (#) are comments and are not processed.

You may define configuration options for a specific log file and place it under the global options. However, it is advisable to create a separate configuration file for any specific log file in the /etc/logrotate.d/ directory and define any configuration options there.

The following is an example of a configuration file placed in the /etc/logrotate.d/ directory:

/var/log/messages {
  rotate 5
  weekly
  postrotate
  /usr/bin/killall -HUP syslogd
  endscript
}

The configuration options in this file are specific for the /var/log/messages log file only. The settings specified here override the global settings where possible. Thus the rotated /var/log/messages log file will be kept for five weeks instead of four weeks as was defined in the global options.

The following is a list of some of the directives you can specify in your logrotate configuration file:

For the full list of directives and various configuration options, see the logrotate(5) manual page.

Under certain circumstances the rsyslog exceeds the limit for a maximum number of open files. Consequently, the rsyslog cannot open new files.

To increase the limit of open files in the rsyslog:

Create the /etc/systemd/system/rsylog.service.d/increase_nofile_limit.conf file with the following content:

[Service]
LimitNOFILE=16384

Leaving special directives aside, rsyslog handles messages as defined by rules that consist of a filter condition and an action to be performed if the condition is true. With a traditionally written /etc/rsyslog.conf file, all rules are evaluated in order of appearance for every input message. This process starts with the first rule and continues until all rules have been processed or until the message is discarded by one of the rules.

However, rules can be grouped into sequences called rulesets. With rulesets, you can limit the effect of certain rules only to selected inputs or enhance the performance of rsyslog by defining a distinct set of actions bound to a specific input. In other words, filter conditions that will be inevitably evaluated as false for certain types of messages can be skipped. The legacy ruleset definition in /etc/rsyslog.conf can look as follows:

$RuleSet rulesetname
rule
rule2

The rule ends when another rule is defined, or the default ruleset is called as follows:

$RuleSet RSYSLOG_DefaultRuleset

With the new configuration format in rsyslog 7, the input() and ruleset() statements are reserved for this operation. The new format ruleset definition in /etc/rsyslog.conf can look as follows:

ruleset(name="rulesetname") {
   rule
   rule2
   call rulesetname2
   …
}

Replace rulesetname with an identifier for your ruleset. The ruleset name cannot start with RSYSLOG_ since this namespace is reserved for use by rsyslog. RSYSLOG_DefaultRuleset then defines the default set of rules to be performed if the message has no other ruleset assigned. With rule and rule2 you can define rules in filter-action format mentioned above. With the call parameter, you can nest rulesets by calling them from inside other ruleset blocks.

After creating a ruleset, you need to specify what input it will apply to:

input(type="input_type" port="port_num" ruleset="rulesetname");

Here you can identify an input message by input_type, which is an input module that gathered the message, or by port_num – the port number. Other parameters such as file or tag can be specified for input(). Replace rulesetname with a name of the ruleset to be evaluated against the message. In case an input message is not explicitly bound to a ruleset, the default ruleset is triggered.

You can also use the legacy format to define rulesets, for more information see the section called “Online Documentation”.

ruleset(name="remote-6514") {
  action(type="omfile" file="/var/log/remote-6514")
}

ruleset(name="remote-601") {
  cron.* action(type="omfile" file="/var/log/remote-601-cron")
  mail.* action(type="omfile" file="/var/log/remote-601-mail")
}

input(type="imtcp" port="6514" ruleset="remote-6514");
input(type="imtcp" port="601" ruleset="remote-601");

The compatibility mode specified via the -c option exists in rsyslog version 5 but not in version 7. Also, the sysklogd-style command-line options are deprecated and configuring rsyslog through these command-line options should be avoided. However, you can use several templates and directives to configure rsyslogd to emulate sysklogd-like behavior.

For more information on various rsyslogd options, see the rsyslogd(8)manual page.

Based on where the messages are stored, there are several types of queues: direct, in-memory, disk, and disk-assisted in-memory queues that are most widely used. You can choose one of these types for the main message queue and also for action queues. Add the following into /etc/rsyslog.conf:

object(queue.type=”queue_type”)

By adding this you can apply the setting for:

Replace queue_type with one of direct, linkedlist or fixedarray (which are in-memory queues), or disk.

The default setting for a main message queue is the FixedArray queue with a limit of 10,000 messages. Action queues are by default set as Direct queues.

Direct Queues

For many simple operations, such as when writing output to a local file, building a queue in front of an action is not needed. To avoid queuing, use:

object(queue.type=”Direct”)

Replace object with main_queue, action or ruleset to use this option to the main message queue, an action queue or for the ruleset respectively. With direct queue, messages are passed directly and immediately from the producer to the consumer.

Disk Queues

Disk queues store messages strictly on a hard drive, which makes them highly reliable but also the slowest of all possible queuing modes. This mode can be used to prevent the loss of highly important log data. However, disk queues are not recommended in most use cases. To set a disk queue, type the following into /etc/rsyslog.conf:

object(queue.type=”Disk”)

Replace object with main_queue, action or ruleset to use this option to the main message queue, an action queue or for the ruleset respectively. The default size of a queue can be modified with the following configuration directive:

object(queue.size=”size”)

where size represents the specified size of disk queue part. The defined size limit is not restrictive, rsyslog always writes one complete queue entry, even if it violates the size limit. Each part of a disk queue matches with an individual file. The naming directive for these files looks as follows:

object(queue.filename=”name”)

This sets a name prefix for the file followed by a 7-digit number starting at one and incremented for each file.

object(queue.maxfilesize=”size”)

Disk queues are written in parts, with a default size 1 MB. Specify size to use a different value.

In-memory Queues

With in-memory queue, the enqueued messages are held in memory which makes the process very fast. The queued data is lost if the computer is power cycled or shut down. However, you can use the action (queue.saveonshutdown=”on”) setting to save the data before shutdown. There are two types of in-memory queues:

In general, use LinkedList queues when in doubt. Compared to FixedArray, it consumes less memory and lowers the processing overhead.

Use the following syntax to configure in-memory queues:

object(queue.type=”LinkedList”)

object(queue.type=”FixedArray”)

Replace object with main_queue, action or ruleset to use this option to the main message queue, an action queue or for the ruleset respectively.

Disk-Assisted In-memory Queues

Both disk and in-memory queues have their advantages and rsyslog lets you combine them in disk-assisted in-memory queues. To do so, configure a normal in-memory queue and then add the queue.filename=”file_name” directive to its block to define a file name for disk assistance. This queue then becomes disk-assisted, which means it couples an in-memory queue with a disk queue to work in tandem.

The disk queue is activated if the in-memory queue is full or needs to persist after shutdown. With a disk-assisted queue, you can set both disk-specific and in-memory specific configuration parameters. This type of queue is probably the most commonly used, it is especially useful for potentially long-running and unreliable actions.

To specify the functioning of a disk-assisted in-memory queue, use the so-called watermarks:

object(queue.highwatermark=”number”)

object(queue.lowwatermark=”number”)

Replace object with main_queue, action or ruleset to use this option to the main message queue, an action queue or for the ruleset respectively. Replace number with a number of enqueued messages. When an in-memory queue reaches the number defined by the high watermark, it starts writing messages to disk and continues until the in-memory queue size drops to the number defined with the low watermark. Correctly set watermarks minimize unnecessary disk writes, but also leave memory space for message bursts since writing to disk files is rather lengthy. Therefore, the high watermark must be lower than the whole queue capacity set with queue.size. The difference between the high watermark and the overall queue size is a spare memory buffer reserved for message bursts. On the other hand, setting the high watermark too low will turn on disk assistance unnecessarily often.

Rsyslog runs as the syslogd daemon and is managed by SELinux. Therefore all files to which rsyslog is required to write to, must have the appropriate SELinux file context.

All types of queues can be further configured to match your requirements. You can use several directives to modify both action queues and the main message queue. Currently, there are more than 20 queue parameters available, see the section called “Online Documentation”. Some of these settings are used commonly, others, such as worker thread management, provide closer control over the queue behavior and are reserved for advanced users. With advanced settings, you can optimize rsyslog's performance, schedule queuing, or modify the behavior of a queue on system shutdown.

Limiting Queue Size

You can limit the number of messages that queue can contain with the following setting:

object(queue.highwatermark=”number”)

Replace object with main_queue, action or ruleset to use this option to the main message queue, an action queue or for the ruleset respectively. Replace number with a number of enqueued messages. You can set the queue size only as the number of messages, not as their actual memory size. The default queue size is 10,000 messages for the main message queue and ruleset queues, and 1000 for action queues.

Disk assisted queues are unlimited by default and cannot be restricted with this directive, but you can reserve them physical disk space in bytes with the following settings:

object(queue.maxdiskspace=”number”)

Replace object with main_queue, action or ruleset. When the size limit specified by number is hit, messages are discarded until sufficient amount of space is freed by dequeued messages.

Discarding Messages

When a queue reaches a certain number of messages, you can discard less important messages in order to save space in the queue for entries of higher priority. The threshold that launches the discarding process can be set with the so-called discard mark:

object(queue.discardmark=”number”)

Replace object with MainMsg or with Action to use this option to the main message queue or for an action queue respectively. Here, number stands for a number of messages that have to be in the queue to start the discarding process. To define which messages to discard, use:

object(queue.discardseverity=”number”)

Replace number with one of the following numbers for respective priorities: 7 (debug), 6 (info), 5 (notice), 4 (warning), 3 (err), 2 (crit), 1 (alert), or 0 (emerg). With this setting, both newly incoming and already queued messages with lower than defined priority are erased from the queue immediately after the discard mark is reached.

Using Timeframes

You can configure rsyslog to process queues during a specific time period. With this option you can, for example, transfer some processing into off-peak hours. To define a time frame, use the following syntax:

object(queue.dequeuetimebegin=”hour”)

object(queue.dequeuetimeend=”hour”)

With hour you can specify hours that bound your time frame. Use the 24-hour format without minutes.

Configuring Worker Threads

A worker thread performs a specified action on the enqueued message. For example, in the main message queue, a worker task is to apply filter logic to each incoming message and enqueue them to the relevant action queues. When a message arrives, a worker thread is started automatically. When the number of messages reaches a certain number, another worker thread is turned on. To specify this number, use:

object(queue.workerthreadminimummessages=”number”)

Replace number with a number of messages that will trigger a supplemental worker thread. For example, with number set to 100, a new worker thread is started when more than 100 messages arrive. When more than 200 messages arrive, the third worker thread starts and so on. However, too many working threads running in parallel becomes ineffective, so you can limit the maximum number of them by using:

object(queue.workerthreads=”number”)

where number stands for a maximum number of working threads that can run in parallel. For the main message queue, the default limit is 1 thread. Once a working thread has been started, it keeps running until an inactivity timeout appears. To set the length of timeout, type:

object(queue.timeoutworkerthreadshutdown=”time”)

Replace time with the duration set in milliseconds. Specifies time without new messages after which the worker thread will be closed. Default setting is one minute.

Batch Dequeuing

To increase performance, you can configure rsyslog to dequeue multiple messages at once. To set the upper limit for such dequeueing, use:

$object(queue.DequeueBatchSize= ”number”)

Replace number with the maximum number of messages that can be dequeued at once. Note that a higher setting combined with a higher number of permitted working threads results in greater memory consumption.

Terminating Queues

When terminating a queue that still contains messages, you can try to minimize the data loss by specifying a time interval for worker threads to finish the queue processing:

object(queue.timeoutshutdown=”time”)

Specify time in milliseconds. If after that period there are still some enqueued messages, workers finish the current data element and then terminate. Unprocessed messages are therefore lost. Another time interval can be set for workers to finish the final element:

object(queue.timeoutactioncompletion=”time”)

In case this timeout expires, any remaining workers are shut down. To save data at shutdown, use:

object(queue.saveonshutdown=”on”)

If set, all queue elements are saved to disk before rsyslog terminates.

In the new syntax available in rsyslog 7, queues are defined inside the action() object that can be used both separately or inside a ruleset in /etc/rsyslog.conf. The format of an action queue is as follows:

action(type="action_type "queue.size="queue_size" queue.type="queue_type" queue.filename="file_name"

Replace action_type with the name of the module that is to perform the action and replace queue_size with a maximum number of messages the queue can contain. For queue_type, choose disk or select from one of the in-memory queues: direct, linkedlist or fixedarray. For file_name specify only a file name, not a path. Note that if creating a new directory to hold log files, the SELinux context must be set. See Section 23.4.2, “Creating a New Directory for rsyslog Log Files” for an example.

action(type="omfile" queue.size="10000" queue.type="linkedlist" queue.filename="logfile")

The rsyslog 7 syntax for a direct action queues is as follows:

. action(type="omfile" file="/var/lib/rsyslog/log_file
   )

The rsyslog 7 syntax for an action queue with multiple parameters can be written as follows:

. action(type="omfile"
       queue.filename="log_file"
       queue.type="linkedlist"
       queue.size="10000"
   )

The default work directory, or the last work directory to be set, will be used. If required to use a different work directory, add a line as follows before the action queue:

global(workDirectory="/directory")

. action(type="omfwd"
   queue.type="linkedlist"
   queue.filename="example_fwd"
   action.resumeRetryCount="-1"
   queue.saveOnShutdown="on"
   target="example.com" port="6514" protocol="tcp"
   )

Rsyslog 7 has a number of different templates styles. The string template most closely resembles the legacy format. Reproducing the templates from the example above using the string format would look as follows:

template(name="TmplAuthpriv" type="string"
     string="/var/log/remote/auth/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log"
    )

template(name="TmplMsg" type="string"
     string="/var/log/remote/msg/%HOSTNAME%/%PROGRAMNAME:::secpath-replace%.log"
    )

These templates can also be written in the list format as follows:

template(name="TmplAuthpriv" type="list") {
  constant(value="/var/log/remote/auth/")
  property(name="hostname")
  constant(value="/")
  property(name="programname" SecurePath="replace")
  constant(value=".log")
  }

template(name="TmplMsg" type="list") {
  constant(value="/var/log/remote/msg/")
  property(name="hostname")
  constant(value="/")
  property(name="programname" SecurePath="replace")
  constant(value=".log")
  }

This template text format might be easier to read for those new to rsyslog and therefore can be easier to adapt as requirements change.

To complete the change to the new syntax, we need to reproduce the module load command, add a rule set, and then bind the rule set to the protocol, port, and ruleset:

module(load="imtcp")

ruleset(name="remote1"){
   authpriv.*  action(type="omfile" DynaFile="TmplAuthpriv")
   *.info;mail.none;authpriv.none;cron.none action(type="omfile" DynaFile="TmplMsg")
}

input(type="imtcp" port="10514" ruleset="remote1")

The Text File Input Module, abbreviated as imfile, enables rsyslog to convert any text file into a stream of syslog messages. You can use imfile to import log messages from applications that create their own text file logs. To load imfile, add the following into /etc/rsyslog.conf:

    module(load=”imfile”
    				PollingInterval=”int”)

It is sufficient to load imfile once, even when importing multiple files. The PollingInterval module argument specifies how often rsyslog checks for changes in connected text files. The default interval is 10 seconds, to change it, replace int with a time interval specified in seconds.

To identify the text files to import, use the following syntax in /etc/rsyslog.conf:

# File 1
input(type="imfile"
   File="path_to_file"
   Tag="tag:"
   Severity="severity"
   Facility="facility")

# File 2
input(type="imfile"
   File="path_to_file2")
...

Settings required to specify an input text file:

Apart from the required directives, there are several other settings that can be applied on the text input. Set the severity of imported messages by replacing severity with an appropriate keyword. Replace facility with a keyword to define the subsystem that produced the message. The keywords for severity and facility are the same as those used in facility/priority-based filters, see Section 23.2.1, “Filters”.

module(load=”imfile”)
input(type="imfile"
   File="/var/log/httpd/error_log"
   Tag="apache-error:")

Processing of log data can be faster and more convenient when performed in a database rather than with text files. Based on the type of DBMS used, choose from various output modules such as ommysql, ompgsql, omoracle, or ommongodb. As an alternative, use the generic omlibdbi output module that relies on the libdbi library. The omlibdbi module supports database systems Firebird/Interbase, MS SQL, Sybase, SQLite, Ingres, Oracle, mSQL, MySQL, and PostgreSQL.

module(load=”ommysql”)

. action(type”ommysql”
server=”database-server”
db=”database-name”
uid=”database-userid”
pwd=”database-password”
serverport=”1234”)

Confidentiality and integrity in network transmissions can be provided by either the TLS or GSSAPI encryption protocol.

Transport Layer Security (TLS) is a cryptographic protocol designed to provide communication security over the network. When using TLS, rsyslog messages are encrypted before sending, and mutual authentication exists between the sender and receiver. For configuring TLS, see the section called “Configuring Encrypted Message Transfer with TLS”.

Generic Security Service API (GSSAPI) is an application programming interface for programs to access security services. To use it in connection with rsyslog you must have a functioning Kerberos environment. For configuring GSSAPI, see the section called “Configuring Encrypted Message Transfer with GSSAPI”.

Configuring Encrypted Message Transfer with TLS

To use encrypted transport through TLS, you need to configure both the server and the client.

Configuring Encrypted Message Transfer with GSSAPI

In rsyslog, interaction with GSSAPI is provided by the imgssapi module. To turn on the GSSAPI transfer mode:

$ModLoad imgssapi
$InputGSSServerPermitPlainTCP on
$InputGSSServerRun 1514

Reliable Event Logging Protocol (RELP) is a networking protocol for data logging in computer networks. It is designed to provide reliable delivery of event messages, which makes it useful in environments where message loss is not acceptable.

Configuring RELP

To configure RELP, you need to configure both the server and the client using the /etc/rsyslog.conf file.

Configuring RELP with TLS

To configure RELP with TLS, you need to configure authentication. Then, you need to configure both the server and the client using the /etc/rsyslog.conf file.

The imjournal module is Rsyslog's input module to natively read the journal files (see Section 23.7, “Interaction of Rsyslog and Journal”). Journal messages are then logged in text format as other rsyslog messages. However, with further processing, it is possible to translate meta data provided by Journal into a structured message.

To import data from Journal to Rsyslog, use the following configuration in /etc/rsyslog.conf:

module(load=”imjournal”
  PersistStateInterval=”number_of_messages”
  StateFile=”path”
  ratelimit.interval=”seconds”
  ratelimit.burst=”burst_number”
  IgnorePreviousMessages=”off/on”)

module(load”imjournal”)
module(load”imuxsock”
  SysSock.Use=”off”
  Socket="/run/systemd/journal/syslog")

You can translate all data and meta data stored by Journal into structured messages. Some of these meta data entries are listed in Example 23.19, “Verbose journalctl Output”, for a complete list of journal fields see the systemd.journal-fields(7) manual page. For example, it is possible to focus on kernel journal fields, that are used by messages originating in the kernel.

To create a lumberjack-formatted message that is required by rsyslog's parsing module, use the following template:

template(name="CEETemplate" type="string" string="%TIMESTAMP% %HOSTNAME% %syslogtag% @cee: %$!all-json%\n")

This template prepends the @cee: string to the JSON string and can be applied, for example, when creating an output file with omfile module. To access JSON field names, use the $! prefix. For example, the following filter condition searches for messages with specific hostname and UID:

($!hostname == "hostname" && $!UID== "UID")

The mmjsonparse module is used for parsing structured messages.

These messages can come from Journal or from other input sources, and must be formatted in a way defined by the Lumberjack project. These messages are identified by the presence of the @cee: string. Then, mmjsonparse checks if the JSON structure is valid and then the message is parsed.

To parse lumberjack-formatted JSON messages with mmjsonparse, use the following configuration in the /etc/rsyslog.conf:

module(load”mmjsonparse”)

. :mmjsonparse:

In this example, the mmjsonparse module is loaded on the first line, then all messages are forwarded to it. Currently, there are no configuration parameters available for mmjsonparse.

Rsyslog supports storing JSON logs in the MongoDB document database through the ommongodb output module.

To forward log messages into MongoDB, use the following syntax in the /etc/rsyslog.conf (configuration parameters for ommongodb are available only in the new configuration format; see Section 23.3, “Using the New Configuration Format”):

module(load”ommongodb”)

. action(type="ommongodb" server="DB_server" serverport="port" db="DB_name" collection="collection_name" uid="UID" pwd="password")

You can shape the form of the final database output with use of templates. By default, rsyslog uses a template based on standard lumberjack field names.

To access the journal logs, use the journalctl tool. For a basic view of the logs type as root:

journalctl

An output of this command is a list of all log files generated on the system including messages generated by system components and by users. The structure of this output is similar to one used in /var/log/messages/ but with certain improvements:

# journalctl
-- Logs begin at Thu 2013-08-01 15:42:12 CEST, end at Thu 2013-08-01 15:48:48 CEST. --
Aug 01 15:42:12 localhost systemd-journal[54]: Allowing runtime journal files to grow to 49.7M.
Aug 01 15:42:12 localhost kernel: Initializing cgroup subsys cpuset
Aug 01 15:42:12 localhost kernel: Initializing cgroup subsys cpu

[...]

In many cases, only the latest entries in the journal log are relevant. The simplest way to reduce journalctl output is to use the -n option that lists only the specified number of most recent log entries:

journalctl -n Number

Replace Number with the number of lines to be shown. When no number is specified, journalctl displays the ten most recent entries.

The journalctl command allows controlling the form of the output with the following syntax:

journalctl -o form

Replace form with a keyword specifying a desired form of output. There are several options, such as verbose, which returns full-structured entry items with all fields, export, which creates a binary stream suitable for backups and network transfer, and json, which formats entries as JSON data structures. For the full list of keywords, see the journalctl(1) manual page.

# journalctl -o verbose
[...]

Fri 2013-08-02 14:41:22 CEST [s=e1021ca1b81e4fc688fad6a3ea21d35b;i=55c;b=78c81449c920439da57da7bd5c56a770;m=27cc
    _BOOT_ID=78c81449c920439da57da7bd5c56a770
    PRIORITY=5
    SYSLOG_FACILITY=3
    _TRANSPORT=syslog
    _MACHINE_ID=69d27b356a94476da859461d3a3bc6fd
    _HOSTNAME=localhost.localdomain
    _PID=562
    _COMM=dbus-daemon
    _EXE=/usr/bin/dbus-daemon
    _CMDLINE=/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
    _SYSTEMD_CGROUP=/system/dbus.service
    _SYSTEMD_UNIT=dbus.service
    SYSLOG_IDENTIFIER=dbus
    SYSLOG_PID=562
    _UID=81
    _GID=81
    _SELINUX_CONTEXT=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023
    MESSAGE=[system] Successfully activated service 'net.reactivated.Fprint'
    _SOURCE_REALTIME_TIMESTAMP=1375447282839181

[...]

By default, Journal users without root privileges can only see log files generated by them. The system administrator can add selected users to the adm group, which grants them access to complete log files. To do so, type as root:

usermod -a -G adm username

Here, replace username with a name of the user to be added to the adm group. This user then receives the same output of the journalctl command as the root user. Note that access control only works when persistent storage is enabled for Journal.

When called without parameters, journalctl shows the full list of entries, starting with the oldest entry collected. With the live view, you can supervise the log messages in real time as new entries are continuously printed as they appear. To start journalctl in live view mode, type:

journalctl -f

This command returns a list of the ten most current log lines. The journalctl utility then stays running and waits for new changes to show them immediately.

The output of the journalctl command executed without parameters is often extensive, therefore you can use various filtering methods to extract information to meet your needs.

Filtering by Priority

Log messages are often used to track erroneous behavior on the system. To view only entries with a selected or higher priority, use the following syntax:

journalctl -p priority

Here, replace priority with one of the following keywords (or with a number): debug (7), info (6), notice (5), warning (4), err (3), crit (2), alert (1), and emerg (0).

journalctl -p err

Filtering by Time

To view log entries only from the current boot, type:

journalctl -b

If you reboot your system just occasionally, the -b will not significantly reduce the output of journalctl. In such cases, time-based filtering is more helpful:

journalctl --since=value --until=value

With --since and --until, you can view only log messages created within a specified time range. You can pass values to these options in form of date or time or both as shown in the following example.

journalctl -p warning --since="2013-3-16 23:59:59"

Advanced Filtering

Example 23.19, “Verbose journalctl Output” lists a set of fields that specify a log entry and can all be used for filtering. For a complete description of meta data that systemd can store, see the systemd.journal-fields(7) manual page. This meta data is collected for each log message, without user intervention. Values are usually text-based, but can take binary and large values; fields can have multiple values assigned though it is not very common.

To view a list of unique values that occur in a specified field, use the following syntax:

journalctl -F fieldname

Replace fieldname with a name of a field you are interested in.

To show only log entries that fit a specific condition, use the following syntax:

journalctl fieldname=value

Replace fieldname with a name of a field and value with a specific value contained in that field. As a result, only lines that match this condition are returned.

journalctl

journalctl fieldname=

You can specify multiple values for one field:

journalctl fieldname=value1 fieldname=value2 ...

Specifying two matches for the same field results in a logical OR combination of the matches. Entries matching value1 or value2 are displayed.

Also, you can specify multiple field-value pairs to further reduce the output set:

journalctl fieldname1=value fieldname2=value ...

If two matches for different field names are specified, they will be combined with a logical AND. Entries have to match both conditions to be shown.

With use of the + symbol, you can set a logical OR combination of matches for multiple fields:

journalctl fieldname1=value + fieldname2=value ...

This command returns entries that match at least one of the conditions, not only those that match both of them.

journalctl _UID=70 _SYSTEMD_UNIT=avahi-daemon.service _SYSTEMD_UNIT=crond.service

You can apply the aforementioned filtering also in the live-view mode to keep track of the latest changes in the selected group of log entries:

journalctl -f fieldname=value ...

By default, Journal stores log files only in memory or a small ring-buffer in the /run/log/journal/ directory. This is sufficient to show recent log history with journalctl. This directory is volatile, log data is not saved permanently. With the default configuration, syslog reads the journal logs and stores them in the /var/log/ directory. With persistent logging enabled, journal files are stored in /var/log/journal which means they persist after reboot. Journal can then replace rsyslog for some users (but see the chapter introduction).

Enabled persistent storage has the following advantages

Persistent storage has also certain disadvantages:

To enable persistent storage for Journal, create the journal directory manually as shown in the following example. As root type:

mkdir -p /var/log/journal/

Then, restart journald to apply the change:

systemctl restart systemd-journald

Most log files are stored in plain text format. You can view them with any text editor such as Vi or Emacs. Some log files are readable by all users on the system; however, root privileges are required to read most log files.

To view system log files in an interactive, real-time application, use the System Log.

~]# yum install gnome-system-log

After you have installed the gnome-system-log package, open the System Log by clicking Applications System Tools System Log, or type the following command at a shell prompt:

~]$ gnome-system-log

The application only displays log files that exist; thus, the list might differ from the one shown in Figure 23.2, “System Log”.

Figure 23.2. System Log

The System Log application lets you filter any existing log file. Click on the button marked with the gear symbol to view the menu, select menu:[Filters > > Manage Filters] to define or edit the desired filter.

Figure 23.3. System Log - Filters

Adding or editing a filter lets you define its parameters as is shown in Figure 23.4, “System Log - defining a filter”.

Figure 23.4. System Log - defining a filter

When defining a filter, the following parameters can be edited:

When you have at least one filter defined, it can be selected from the Filters menu and it will automatically search for the strings you have defined in the filter and highlight or hide every successful match in the log file you are currently viewing.

Figure 23.5. System Log - enabling a filter

When you select the Show matches only option, only the matched strings will be shown in the log file you are currently viewing.

To add a log file you want to view in the list, select File Open . This will display the Open Log window where you can select the directory and file name of the log file you want to view. Figure 23.6, “System Log - adding a log file” illustrates the Open Log window.

Figure 23.6. System Log - adding a log file

Click on the Open button to open the file. The file is immediately added to the viewing list where you can select it and view its contents.

System Log monitors all opened logs by default. If a new line is added to a monitored log file, the log name appears in bold in the log list. If the log file is selected or displayed, the new lines appear in bold at the bottom of the log file. Figure 23.7, “System Log - new log alert” illustrates a new alert in the cron log file and in the messages log file. Clicking on the messages log file displays the logs in the file with the new lines in bold.

Figure 23.7. System Log - new log alert