Search

Chapter 9. Configuring logging

download PDF

Most services in Red Hat Enterprise Linux log status messages, warnings, and errors. You can use the rsyslogd service to log these entries to local files or to a remote logging server.

9.1. Configuring a remote logging solution

To ensure that logs from various machines in your environment are recorded centrally on a logging server, you can configure the Rsyslog application to record logs that fit specific criteria from the client system to the server.

9.1.1. The Rsyslog logging service

The Rsyslog application, in combination with the systemd-journald service, provides local and remote logging support in Red Hat Enterprise Linux. The rsyslogd daemon continuously reads syslog messages received by the systemd-journald service from the Journal. rsyslogd then filters and processes these syslog events and records them to rsyslog log files or forwards them to other services according to its configuration.

The rsyslogd daemon also provides extended filtering, encryption protected relaying of messages, input and output modules, and support for transportation using the TCP and UDP protocols.

In /etc/rsyslog.conf, which is the main configuration file for rsyslog, you can specify the rules according to which rsyslogd handles the messages. Generally, you can classify messages by their source and topic (facility) and urgency (priority), and then assign an action that should be performed when a message fits these criteria.

In /etc/rsyslog.conf, you can also see a list of log files maintained by rsyslogd. Most log files are located in the /var/log/ directory. Some applications, such as httpd and samba, store their log files in a subdirectory within /var/log/.

Additional resources

  • The rsyslogd(8) and rsyslog.conf(5) man pages.
  • Documentation installed with the rsyslog-doc package in the /usr/share/doc/rsyslog/html/index.html file.

9.1.2. Installing Rsyslog documentation

The Rsyslog application has extensive online documentation that is available at https://www.rsyslog.com/doc/, but you can also install the rsyslog-doc documentation package locally.

Prerequisites

  • You have activated the AppStream repository on your system.
  • You are authorized to install new packages using sudo.

Procedure

  • Install the rsyslog-doc package:

    # yum install rsyslog-doc

Verification

  • Open the /usr/share/doc/rsyslog/html/index.html file in a browser of your choice, for example:

    $ firefox /usr/share/doc/rsyslog/html/index.html &

9.1.3. Configuring a server for remote logging over TCP

The Rsyslog application enables you to both run a logging server and configure individual systems to send their log files to the logging server. To use remote logging through TCP, configure both the server and the client. The server collects and analyzes the logs sent by one or more client systems.

With the Rsyslog application, you can maintain a centralized logging system where log messages are forwarded to a server over the network. To avoid message loss when the server is not available, you can configure an action queue for the forwarding action. This way, messages that failed to be sent are stored locally until the server is reachable again. Note that such queues cannot be configured for connections using the UDP protocol.

The omfwd plug-in provides forwarding over UDP or TCP. The default protocol is UDP. Because the plug-in is built in, it does not have to be loaded.

By default, rsyslog uses TCP on port 514.

Prerequisites

  • Rsyslog is installed on the server system.
  • You are logged in as root on the server.
  • The policycoreutils-python-utils package is installed for the optional step using the semanage command.
  • The firewalld service is running.

Procedure

  1. Optional: To use a different port for rsyslog traffic, add the syslogd_port_t SELinux type to port. For example, enable port 30514:

    # semanage port -a -t syslogd_port_t -p tcp 30514
  2. Optional: To use a different port for rsyslog traffic, configure firewalld to allow incoming rsyslog traffic on that port. For example, allow TCP traffic on port 30514:

    # firewall-cmd --zone=<zone-name> --permanent --add-port=30514/tcp
    success
    # firewall-cmd --reload
  3. Create a new file in the /etc/rsyslog.d/ directory named, for example, remotelog.conf, and insert the following content:

    # Define templates before the rules that use them
    # Per-Host templates for remote systems
    template(name="TmplAuthpriv" type="list") {
        constant(value="/var/log/remote/auth/")
        property(name="hostname")
        constant(value="/")
        property(name="programname" SecurePath="replace")
        constant(value=".log")
        }
    
    template(name="TmplMsg" type="list") {
        constant(value="/var/log/remote/msg/")
        property(name="hostname")
        constant(value="/")
        property(name="programname" SecurePath="replace")
        constant(value=".log")
        }
    
    # Provides TCP syslog reception
    module(load="imtcp")
    
    # Adding this ruleset to process remote messages
    ruleset(name="remote1"){
         authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
          *.info;mail.none;authpriv.none;cron.none
    action(type="omfile" DynaFile="TmplMsg")
    }
    
    input(type="imtcp" port="30514" ruleset="remote1")
  4. Save the changes to the /etc/rsyslog.d/remotelog.conf file.
  5. Test the syntax of the /etc/rsyslog.conf file:

    # rsyslogd -N 1
    rsyslogd: version 8.1911.0-2.el8, config validation run...
    rsyslogd: End of config validation run. Bye.
  6. Make sure the rsyslog service is running and enabled on the logging server:

    # systemctl status rsyslog
  7. Restart the rsyslog service.

    # systemctl restart rsyslog
  8. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

    # systemctl enable rsyslog

Your log server is now configured to receive and store log files from the other systems in your environment.

Additional resources

  • rsyslogd(8), rsyslog.conf(5), semanage(8), and firewall-cmd(1) man pages.
  • Documentation installed with the rsyslog-doc package in the /usr/share/doc/rsyslog/html/index.html file.

9.1.4. Configuring remote logging to a server over TCP

You can configure a system for forwarding log messages to a server over the TCP protocol. The omfwd plug-in provides forwarding over UDP or TCP. The default protocol is UDP. Because the plug-in is built in, you do not have to load it.

Prerequisites

  • The rsyslog package is installed on the client systems that should report to the server.
  • You have configured the server for remote logging.
  • The specified port is permitted in SELinux and open in firewall.
  • The system contains the policycoreutils-python-utils package, which provides the semanage command for adding a non-standard port to the SELinux configuration.

Procedure

  1. Create a new file in the /etc/rsyslog.d/ directory named, for example, 10-remotelog.conf, and insert the following content:

    *.* action(type="omfwd"
          queue.type="linkedlist"
          queue.filename="example_fwd"
          action.resumeRetryCount="-1"
          queue.saveOnShutdown="on"
          target="example.com" port="30514" protocol="tcp"
         )

    Where:

    • The queue.type="linkedlist" setting enables a LinkedList in-memory queue,
    • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
    • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
    • The queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.
    • The last line forwards all received messages to the logging server. Port specification is optional.

      With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

    Note

    Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

  2. Restart the rsyslog service.

    # systemctl restart rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message:

    # logger test
  2. On the server system, view the /var/log/messages log, for example:

    # cat /var/log/remote/msg/hostname/root.log
    Feb 25 03:53:17 hostname root[6064]: test

    Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • rsyslogd(8) and rsyslog.conf(5) man pages.
  • Documentation installed with the rsyslog-doc package in the /usr/share/doc/rsyslog/html/index.html file.

9.1.5. Configuring TLS-encrypted remote logging

By default, Rsyslog sends remote-logging communication in the plain text format. If your scenario requires to secure this communication channel, you can encrypt it using TLS.

To use encrypted transport through TLS, configure both the server and the client. The server collects and analyzes the logs sent by one or more client systems.

You can use either the ossl network stream driver (OpenSSL) or the gtls stream driver (GnuTLS).

Note

If you have a separate system with higher security, for example, a system that is not connected to any network or has stricter authorizations, use the separate system as the certifying authority (CA).

You can customize your connection settings with stream drivers on the server side on the global, module, and input levels, and on the client side on the global and action levels. The more specific configuration overrides the more general configuration. This means, for example, that you can use ossl in global settings for most connections and gtls on the input and action settings only for specific connections.

Prerequisites

  • You have root access to both the client and server systems.
  • The following packages are installed on the server and the client systems:

    • The rsyslog package.
    • For the ossl network stream driver, the rsyslog-openssl package.
    • For the gtls network stream driver, the rsyslog-gnutls package.
    • For generating certificates by using the certtool command, the gnutls-utils package.
  • On your logging server, the following certificates are in the /etc/pki/ca-trust/source/anchors/ directory and your system configuration is updated by using the update-ca-trust command:

    • ca-cert.pem - a CA certificate that can verify keys and certificates on logging servers and clients.
    • server-cert.pem - a public key of the logging server.
    • server-key.pem - a private key of the logging server.
  • On your logging clients, the following certificates are in the /etc/pki/ca-trust/source/anchors/ directory and your system configuration is updated by using update-ca-trust:

    • ca-cert.pem - a CA certificate that can verify keys and certificates on logging servers and clients.
    • client-cert.pem - a public key of a client.
    • client-key.pem - a private key of a client.

Procedure

  1. Configure the server for receiving encrypted logs from your client systems:

    1. Create a new file in the /etc/rsyslog.d/ directory named, for example, securelogser.conf.
    2. To encrypt the communication, the configuration file must contain paths to certificate files on your server, a selected authentication method, and a stream driver that supports TLS encryption. Add the following lines to the /etc/rsyslog.d/securelogser.conf file:

      # Set certificate files
      global(
        DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
        DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/server-cert.pem"
        DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/server-key.pem"
      )
      
      # TCP listener
      module(
        load="imtcp"
        PermittedPeer=["client1.example.com", "client2.example.com"]
        StreamDriver.AuthMode="x509/name"
        StreamDriver.Mode="1"
        StreamDriver.Name="ossl"
      )
      
      # Start up listener at port 514
      input(
        type="imtcp"
        port="514"
      )
      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option. See the documentation installed with the rsyslog-doc package for more information about less strict authentication modes than x509/name.

    3. Save the changes to the /etc/rsyslog.d/securelogser.conf file.
    4. Verify the syntax of the /etc/rsyslog.conf file and any files in the /etc/rsyslog.d/ directory:

      # rsyslogd -N 1
      rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
      rsyslogd: End of config validation run. Bye.
    5. Make sure the rsyslog service is running and enabled on the logging server:

      # systemctl status rsyslog
    6. Restart the rsyslog service:

      # systemctl restart rsyslog
    7. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog
  2. Configure clients for sending encrypted logs to the server:

    1. On a client system, create a new file in the /etc/rsyslog.d/ directory named, for example, securelogcli.conf.
    2. Add the following lines to the /etc/rsyslog.d/securelogcli.conf file:

      # Set certificate files
      global(
        DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
        DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/client-cert.pem"
        DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/client-key.pem"
      )
      
      
      # Set up the action for all messages
      *.* action(
        type="omfwd"
        StreamDriver="ossl"
        StreamDriverMode="1"
        StreamDriverPermittedPeers="server.example.com"
        StreamDriverAuthMode="x509/name"
        target="server.example.com" port="514" protocol="tcp"
      )
      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option.

    3. Save the changes to the /etc/rsyslog.d/securelogcli.conf file.
    4. Verify the syntax of the /etc/rsyslog.conf file and other files in the /etc/rsyslog.d/ directory:

      # rsyslogd -N 1
      rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
      rsyslogd: End of config validation run. Bye.
    5. Make sure the rsyslog service is running and enabled on the logging server:

      # systemctl status rsyslog
    6. Restart the rsyslog service:

      # systemctl restart rsyslog
    7. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message:

    # logger test
  2. On the server system, view the /var/log/messages log, for example:

    # cat /var/log/remote/msg/<hostname>/root.log
    Feb 25 03:53:17 <hostname> root[6064]: test

    Where <hostname> is the hostname of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • certtool(1), openssl(1), update-ca-trust(8), rsyslogd(8), and rsyslog.conf(5) man pages.
  • Documentation installed with the rsyslog-doc package at /usr/share/doc/rsyslog/html/index.html.
  • Using the logging system role with TLS.

9.1.6. Configuring a server for receiving remote logging information over UDP

The Rsyslog application enables you to configure a system to receive logging information from remote systems. To use remote logging through UDP, configure both the server and the client. The receiving server collects and analyzes the logs sent by one or more client systems. By default, rsyslog uses UDP on port 514 to receive log information from remote systems.

Follow this procedure to configure a server for collecting and analyzing logs sent by one or more client systems over the UDP protocol.

Prerequisites

  • Rsyslog is installed on the server system.
  • You are logged in as root on the server.
  • The policycoreutils-python-utils package is installed for the optional step using the semanage command.
  • The firewalld service is running.

Procedure

  1. Optional: To use a different port for rsyslog traffic than the default port 514:

    1. Add the syslogd_port_t SELinux type to the SELinux policy configuration, replacing portno with the port number you want rsyslog to use:

      # semanage port -a -t syslogd_port_t -p udp portno
    2. Configure firewalld to allow incoming rsyslog traffic, replacing portno with the port number and zone with the zone you want rsyslog to use:

      # firewall-cmd --zone=zone --permanent --add-port=portno/udp
      success
      # firewall-cmd --reload
    3. Reload the firewall rules:

      # firewall-cmd --reload
  2. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, remotelogserv.conf, and insert the following content:

    # Define templates before the rules that use them
    # Per-Host templates for remote systems
    template(name="TmplAuthpriv" type="list") {
        constant(value="/var/log/remote/auth/")
        property(name="hostname")
        constant(value="/")
        property(name="programname" SecurePath="replace")
        constant(value=".log")
        }
    
    template(name="TmplMsg" type="list") {
        constant(value="/var/log/remote/msg/")
        property(name="hostname")
        constant(value="/")
        property(name="programname" SecurePath="replace")
        constant(value=".log")
        }
    
    # Provides UDP syslog reception
    module(load="imudp")
    
    # This ruleset processes remote messages
    ruleset(name="remote1"){
         authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
          *.info;mail.none;authpriv.none;cron.none
    action(type="omfile" DynaFile="TmplMsg")
    }
    
    input(type="imudp" port="514" ruleset="remote1")

    Where 514 is the port number rsyslog uses by default. You can specify a different port instead.

  3. Verify the syntax of the /etc/rsyslog.conf file and all .conf files in the /etc/rsyslog.d/ directory:

    # rsyslogd -N 1
    rsyslogd: version 8.1911.0-2.el8, config validation run...
  4. Restart the rsyslog service.

    # systemctl restart rsyslog
  5. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

    # systemctl enable rsyslog

Additional resources

  • rsyslogd(8) , rsyslog.conf(5), semanage(8), and firewall-cmd(1) man pages.
  • Documentation installed with the rsyslog-doc package in the /usr/share/doc/rsyslog/html/index.html file.

9.1.7. Configuring remote logging to a server over UDP

You can configure a system for forwarding log messages to a server over the UDP protocol. The omfwd plug-in provides forwarding over UDP or TCP. The default protocol is UDP. Because the plug-in is built in, you do not have to load it.

Prerequisites

Procedure

  1. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, 10-remotelogcli.conf, and insert the following content:

    *.* action(type="omfwd"
          queue.type="linkedlist"
          queue.filename="example_fwd"
          action.resumeRetryCount="-1"
          queue.saveOnShutdown="on"
          target="example.com" port="portno" protocol="udp"
         )

    Where:

    • The queue.type="linkedlist" setting enables a LinkedList in-memory queue.
    • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
    • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if the server is not responding.
    • The enabled queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.
    • The portno value is the port number you want rsyslog to use. The default value is 514.
    • The last line forwards all received messages to the logging server, port specification is optional.

      With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

    Note

    Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

  2. Restart the rsyslog service.

    # systemctl restart rsyslog
  3. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

    # systemctl enable rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message:

    # logger test
  2. On the server system, view the /var/log/remote/msg/hostname/root.log log, for example:

    # cat /var/log/remote/msg/hostname/root.log
    Feb 25 03:53:17 hostname root[6064]: test

    Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • rsyslogd(8) and rsyslog.conf(5) man pages.
  • Documentation installed with the rsyslog-doc package at /usr/share/doc/rsyslog/html/index.html.

9.1.8. Load balancing helper in Rsyslog

The RebindInterval setting specifies an interval at which the current connection is broken and is re-established. This setting applies to TCP, UDP, and RELP traffic. The load balancers perceive it as a new connection and forward the messages to another physical target system.

The RebindInterval setting proves to be helpful in scenarios when a target system has changed its IP address. The Rsyslog application caches the IP address when the connection establishes, therefore, the messages are sent to the same server. If the IP address changes, the UDP packets will be lost until the Rsyslog service restarts. Re-establishing the connection will ensure the IP to be resolved by DNS again.

action(type=”omfwd” protocol=”tcp” RebindInterval=”250” target=”example.com” port=”514” …)

action(type=”omfwd” protocol=”udp” RebindInterval=”250” target=”example.com” port=”514” …)

action(type=”omrelp” RebindInterval=”250” target=”example.com” port=”6514” …)

9.1.9. Configuring reliable remote logging

With the Reliable Event Logging Protocol (RELP), you can send and receive syslog messages over TCP with a much reduced risk of message loss. RELP provides reliable delivery of event messages, which makes it useful in environments where message loss is not acceptable. To use RELP, configure the imrelp input module, which runs on the server and receives the logs, and the omrelp output module, which runs on the client and sends logs to the logging server.

Prerequisites

  • You have installed the rsyslog, librelp, and rsyslog-relp packages on the server and the client systems.
  • The specified port is permitted in SELinux and open in the firewall.

Procedure

  1. Configure the client system for reliable remote logging:

    1. On the client system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpclient.conf, and insert the following content:

      module(load="omrelp")
      *.* action(type="omrelp" target="_target_IP_" port="_target_port_")

      Where:

      • target_IP is the IP address of the logging server.
      • target_port is the port of the logging server.
    2. Save the changes to the /etc/rsyslog.d/relpclient.conf file.
    3. Restart the rsyslog service.

      # systemctl restart rsyslog
    4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog
  2. Configure the server system for reliable remote logging:

    1. On the server system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpserv.conf, and insert the following content:

      ruleset(name="relp"){
      *.* action(type="omfile" file="_log_path_")
      }
      
      
      module(load="imrelp")
      input(type="imrelp" port="_target_port_" ruleset="relp")

      Where:

      • log_path specifies the path for storing messages.
      • target_port is the port of the logging server. Use the same value as in the client configuration file.
    2. Save the changes to the /etc/rsyslog.d/relpserv.conf file.
    3. Restart the rsyslog service.

      # systemctl restart rsyslog
    4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message:

    # logger test
  2. On the server system, view the log at the specified log_path, for example:

    # cat /var/log/remote/msg/hostname/root.log
    Feb 25 03:53:17 hostname root[6064]: test

    Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • rsyslogd(8) and rsyslog.conf(5) man pages.
  • Documentation installed with the rsyslog-doc package in the /usr/share/doc/rsyslog/html/index.html file.

9.1.10. Supported Rsyslog modules

To expand the functionality of the Rsyslog application, you can use specific modules. Modules provide additional inputs (Input Modules), outputs (Output Modules), and other functionalities. A module can also provide additional configuration directives that become available after you load the module.

You can list the input and output modules installed on your system by entering the following command:

# ls /usr/lib64/rsyslog/{i,o}m*

You can view the list of all available rsyslog modules in the /usr/share/doc/rsyslog/html/configuration/modules/idx_output.html file after you install the rsyslog-doc package.

9.1.11. Configuring the netconsole service to log kernel messages to a remote host

When logging to disk or using a serial console is not possible, you can use the netconsole kernel module and the same-named service to log kernel messages over a network to a remote rsyslog service.

Prerequisites

  • A system log service, such as rsyslog is installed on the remote host.
  • The remote system log service is configured to receive incoming log entries from this host.

Procedure

  1. Install the netconsole-service package:

    # yum install netconsole-service
  2. Edit the /etc/sysconfig/netconsole file and set the SYSLOGADDR parameter to the IP address of the remote host:

    # SYSLOGADDR=192.0.2.1
  3. Enable and start the netconsole service:

    # systemctl enable --now netconsole

Verification

  • Display the /var/log/messages file on the remote system log server.

9.1.12. Additional resources

9.2. Using the logging system role

As a system administrator, you can use the logging system role to configure a Red Hat Enterprise Linux host as a logging server to collect logs from many client systems.

9.2.1. The logging RHEL system role

With the logging RHEL system role, you can deploy logging configurations on local and remote hosts.

Logging solutions provide multiple ways of reading logs and multiple logging outputs.

For example, a logging system can receive the following inputs:

  • Local files
  • systemd/journal
  • Another logging system over the network

In addition, a logging system can have the following outputs:

  • Logs stored in the local files in the /var/log directory
  • Logs sent to Elasticsearch
  • Logs forwarded to another logging system

With the logging RHEL system role, you can combine the inputs and outputs to fit your scenario. For example, you can configure a logging solution that stores inputs from journal in a local file, whereas inputs read from files are both forwarded to another logging system and stored in the local log files.

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.logging/README.md file
  • /usr/share/doc/rhel-system-roles/logging/ directory
  • RHEL system roles

9.2.2. Applying a local logging RHEL system role

Prepare and apply an Ansible playbook to configure a logging solution on a set of separate machines. Each machine records logs locally.

Prerequisites

Note

You do not have to have the rsyslog package installed, because the RHEL system role installs rsyslog when deployed.

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    ---
    - name: Deploying basics input and implicit files output
      hosts: managed-node-01.example.com
      roles:
        - rhel-system-roles.logging
      vars:
        logging_inputs:
          - name: system_input
            type: basics
        logging_outputs:
          - name: files_output
            type: files
        logging_flows:
          - name: flow1
            inputs: [system_input]
            outputs: [files_output]
  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Verification

  1. Test the syntax of the /etc/rsyslog.conf file:

    # rsyslogd -N 1
    rsyslogd: version 8.1911.0-6.el8, config validation run...
    rsyslogd: End of config validation run. Bye.
  2. Verify that the system sends messages to the log:

    1. Send a test message:

      # logger test
    2. View the /var/log/messages log, for example:

      # cat /var/log/messages
      Aug  5 13:48:31 <hostname> root[6778]: test

      Where <hostname> is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.logging/README.md file
  • /usr/share/doc/rhel-system-roles/logging/ directory

9.2.3. Filtering logs in a local logging RHEL system role

You can deploy a logging solution which filters the logs based on the rsyslog property-based filter.

Prerequisites

Note

You do not have to have the rsyslog package installed, because the RHEL system role installs rsyslog when deployed.

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    ---
    - name: Deploying files input and configured files output
      hosts: managed-node-01.example.com
      roles:
        - rhel-system-roles.logging
      vars:
        logging_inputs:
          - name: files_input
            type: basics
        logging_outputs:
          - name: files_output0
            type: files
            property: msg
            property_op: contains
            property_value: error
            path: /var/log/errors.log
          - name: files_output1
            type: files
            property: msg
            property_op: "!contains"
            property_value: error
            path: /var/log/others.log
        logging_flows:
          - name: flow0
            inputs: [files_input]
            outputs: [files_output0, files_output1]

    Using this configuration, all messages that contain the error string are logged in /var/log/errors.log, and all other messages are logged in /var/log/others.log.

    You can replace the error property value with the string by which you want to filter.

    You can modify the variables according to your preferences.

  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Verification

  1. Test the syntax of the /etc/rsyslog.conf file:

    # rsyslogd -N 1
    rsyslogd: version 8.1911.0-6.el8, config validation run...
    rsyslogd: End of config validation run. Bye.
  2. Verify that the system sends messages that contain the error string to the log:

    1. Send a test message:

      # logger error
    2. View the /var/log/errors.log log, for example:

      # cat /var/log/errors.log
      Aug  5 13:48:31 hostname root[6778]: error

      Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.logging/README.md file
  • /usr/share/doc/rhel-system-roles/logging/ directory

9.2.4. Applying a remote logging solution by using the logging RHEL system role

Follow these steps to prepare and apply a Red Hat Ansible Core playbook to configure a remote logging solution. In this playbook, one or more clients take logs from systemd-journal and forward them to a remote server. The server receives remote input from remote_rsyslog and remote_files and outputs the logs to local files in directories named by remote host names.

Prerequisites

Note

You do not have to have the rsyslog package installed, because the RHEL system role installs rsyslog when deployed.

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    ---
    - name: Deploying remote input and remote_files output
      hosts: managed-node-01.example.com
      roles:
        - rhel-system-roles.logging
      vars:
        logging_inputs:
          - name: remote_udp_input
            type: remote
            udp_ports: [ 601 ]
          - name: remote_tcp_input
            type: remote
            tcp_ports: [ 601 ]
        logging_outputs:
          - name: remote_files_output
            type: remote_files
        logging_flows:
          - name: flow_0
            inputs: [remote_udp_input, remote_tcp_input]
            outputs: [remote_files_output]
    
    - name: Deploying basics input and forwards output
      hosts: managed-node-02.example.com
      roles:
        - rhel-system-roles.logging
      vars:
        logging_inputs:
          - name: basic_input
            type: basics
        logging_outputs:
          - name: forward_output0
            type: forwards
            severity: info
            target: <host1.example.com>
            udp_port: 601
          - name: forward_output1
            type: forwards
            facility: mail
            target: <host1.example.com>
            tcp_port: 601
        logging_flows:
          - name: flows0
            inputs: [basic_input]
            outputs: [forward_output0, forward_output1]
    
    [basic_input]
    [forward_output0, forward_output1]

    Where <host1.example.com> is the logging server.

    Note

    You can modify the parameters in the playbook to fit your needs.

    Warning

    The logging solution works only with the ports defined in the SELinux policy of the server or client system and open in the firewall. The default SELinux policy includes ports 601, 514, 6514, 10514, and 20514. To use a different port, modify the SELinux policy on the client and server systems.

  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Verification

  1. On both the client and the server system, test the syntax of the /etc/rsyslog.conf file:

    # rsyslogd -N 1
    rsyslogd: version 8.1911.0-6.el8, config validation run (level 1), master config /etc/rsyslog.conf
    rsyslogd: End of config validation run. Bye.
  2. Verify that the client system sends messages to the server:

    1. On the client system, send a test message:

      # logger test
    2. On the server system, view the /var/log/<host2.example.com>/messages log, for example:

      # cat /var/log/<host2.example.com>/messages
      Aug  5 13:48:31 <host2.example.com> root[6778]: test

      Where <host2.example.com> is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.logging/README.md file
  • /usr/share/doc/rhel-system-roles/logging/ directory

9.2.5. Using the logging RHEL system role with TLS

Transport Layer Security (TLS) is a cryptographic protocol designed to allow secure communication over the computer network.

As an administrator, you can use the logging RHEL system role to configure a secure transfer of logs using Red Hat Ansible Automation Platform.

9.2.5.1. Configuring client logging with TLS

You can use an Ansible playbook with the logging RHEL system role to configure logging on RHEL clients and transfer logs to a remote logging system using TLS encryption.

This procedure creates a private key and certificate, and configures TLS on all hosts in the clients group in the Ansible inventory. The TLS protocol encrypts the message transmission for secure transfer of logs over the network.

Note

You do not have to call the certificate RHEL system role in the playbook to create the certificate. The logging RHEL system role calls it automatically.

In order for the CA to be able to sign the created certificate, the managed nodes must be enrolled in an IdM domain.

Prerequisites

  • You have prepared the control node and the managed nodes
  • You are logged in to the control node as a user who can run playbooks on the managed nodes.
  • The account you use to connect to the managed nodes has sudo permissions on them.
  • The managed nodes are enrolled in an IdM domain.

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    ---
    - name: Deploying files input and forwards output with certs
      hosts: managed-node-01.example.com
      roles:
        - rhel-system-roles.logging
      vars:
        logging_certificates:
          - name: logging_cert
            dns: ['localhost', 'www.example.com']
            ca: ipa
        logging_pki_files:
          - ca_cert: /local/path/to/ca_cert.pem
            cert: /local/path/to/logging_cert.pem
            private_key: /local/path/to/logging_cert.pem
        logging_inputs:
          - name: input_name
            type: files
            input_log_path: /var/log/containers/*.log
        logging_outputs:
          - name: output_name
            type: forwards
            target: your_target_host
            tcp_port: 514
            tls: true
            pki_authmode: x509/name
            permitted_server: 'server.example.com'
        logging_flows:
          - name: flow_name
            inputs: [input_name]
            outputs: [output_name]

    The playbook uses the following parameters:

    logging_certificates
    The value of this parameter is passed on to certificate_requests in the certificate RHEL system role and used to create a private key and certificate.
    logging_pki_files

    Using this parameter, you can configure the paths and other settings that logging uses to find the CA, certificate, and key files used for TLS, specified with one or more of the following sub-parameters: ca_cert, ca_cert_src, cert, cert_src, private_key, private_key_src, and tls.

    Note

    If you are using logging_certificates to create the files on the target node, do not use ca_cert_src, cert_src, and private_key_src, which are used to copy files not created by logging_certificates.

    ca_cert
    Represents the path to the CA certificate file on the target node. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    cert
    Represents the path to the certificate file on the target node. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    private_key
    Represents the path to the private key file on the target node. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    ca_cert_src
    Represents the path to the CA certificate file on the control node which is copied to the target host to the location specified by ca_cert. Do not use this if using logging_certificates.
    cert_src
    Represents the path to a certificate file on the control node which is copied to the target host to the location specified by cert. Do not use this if using logging_certificates.
    private_key_src
    Represents the path to a private key file on the control node which is copied to the target host to the location specified by private_key. Do not use this if using logging_certificates.
    tls
    Setting this parameter to true ensures secure transfer of logs over the network. If you do not want a secure wrapper, you can set tls: false.
  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Additional resources

9.2.5.2. Configuring server logging with TLS

You can use an Ansible playbook with the logging RHEL system role to configure logging on RHEL servers and set them to receive logs from a remote logging system using TLS encryption.

This procedure creates a private key and certificate, and configures TLS on all hosts in the server group in the Ansible inventory.

Note

You do not have to call the certificate RHEL system role in the playbook to create the certificate. The logging RHEL system role calls it automatically.

In order for the CA to be able to sign the created certificate, the managed nodes must be enrolled in an IdM domain.

Prerequisites

  • You have prepared the control node and the managed nodes
  • You are logged in to the control node as a user who can run playbooks on the managed nodes.
  • The account you use to connect to the managed nodes has sudo permissions on them.
  • The managed nodes are enrolled in an IdM domain.

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    ---
    - name: Deploying remote input and remote_files output with certs
      hosts: managed-node-01.example.com
      roles:
        - rhel-system-roles.logging
      vars:
        logging_certificates:
          - name: logging_cert
            dns: ['localhost', 'www.example.com']
            ca: ipa
        logging_pki_files:
          - ca_cert: /local/path/to/ca_cert.pem
            cert: /local/path/to/logging_cert.pem
            private_key: /local/path/to/logging_cert.pem
        logging_inputs:
          - name: input_name
            type: remote
            tcp_ports: 514
            tls: true
            permitted_clients: ['clients.example.com']
        logging_outputs:
          - name: output_name
            type: remote_files
            remote_log_path: /var/log/remote/%FROMHOST%/%PROGRAMNAME:::secpath-replace%.log
            async_writing: true
            client_count: 20
            io_buffer_size: 8192
        logging_flows:
          - name: flow_name
            inputs: [input_name]
            outputs: [output_name]

    The playbook uses the following parameters:

    logging_certificates
    The value of this parameter is passed on to certificate_requests in the certificate RHEL system role and used to create a private key and certificate.
    logging_pki_files

    Using this parameter, you can configure the paths and other settings that logging uses to find the CA, certificate, and key files used for TLS, specified with one or more of the following sub-parameters: ca_cert, ca_cert_src, cert, cert_src, private_key, private_key_src, and tls.

    Note

    If you are using logging_certificates to create the files on the target node, do not use ca_cert_src, cert_src, and private_key_src, which are used to copy files not created by logging_certificates.

    ca_cert
    Represents the path to the CA certificate file on the target node. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    cert
    Represents the path to the certificate file on the target node. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    private_key
    Represents the path to the private key file on the target node. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    ca_cert_src
    Represents the path to the CA certificate file on the control node which is copied to the target host to the location specified by ca_cert. Do not use this if using logging_certificates.
    cert_src
    Represents the path to a certificate file on the control node which is copied to the target host to the location specified by cert. Do not use this if using logging_certificates.
    private_key_src
    Represents the path to a private key file on the control node which is copied to the target host to the location specified by private_key. Do not use this if using logging_certificates.
    tls
    Setting this parameter to true ensures secure transfer of logs over the network. If you do not want a secure wrapper, you can set tls: false.
  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Additional resources

9.2.6. Using the logging RHEL system roles with RELP

Reliable Event Logging Protocol (RELP) is a networking protocol for data and message logging over the TCP network. It ensures reliable delivery of event messages and you can use it in environments that do not tolerate any message loss.

The RELP sender transfers log entries in form of commands and the receiver acknowledges them once they are processed. To ensure consistency, RELP stores the transaction number to each transferred command for any kind of message recovery.

You can consider a remote logging system in between the RELP Client and RELP Server. The RELP Client transfers the logs to the remote logging system and the RELP Server receives all the logs sent by the remote logging system.

Administrators can use the logging RHEL system role to configure the logging system to reliably send and receive log entries.

9.2.6.1. Configuring client logging with RELP

You can use the logging RHEL system role to configure logging in RHEL systems that are logged on a local machine and can transfer logs to the remote logging system with RELP by running an Ansible playbook.

This procedure configures RELP on all hosts in the clients group in the Ansible inventory. The RELP configuration uses Transport Layer Security (TLS) to encrypt the message transmission for secure transfer of logs over the network.

Prerequisites

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    ---
    - name: Deploying basic input and relp output
      hosts: managed-node-01.example.com
      roles:
        - rhel-system-roles.logging
      vars:
        logging_inputs:
          - name: basic_input
            type: basics
        logging_outputs:
          - name: relp_client
            type: relp
            target: logging.server.com
            port: 20514
            tls: true
            ca_cert: /etc/pki/tls/certs/ca.pem
            cert: /etc/pki/tls/certs/client-cert.pem
            private_key: /etc/pki/tls/private/client-key.pem
            pki_authmode: name
            permitted_servers:
              - '*.server.example.com'
        logging_flows:
          - name: example_flow
            inputs: [basic_input]
            outputs: [relp_client]

    The playbook uses following settings:

    target
    This is a required parameter that specifies the host name where the remote logging system is running.
    port
    Port number the remote logging system is listening.
    tls

    Ensures secure transfer of logs over the network. If you do not want a secure wrapper you can set the tls variable to false. By default tls parameter is set to true while working with RELP and requires key/certificates and triplets {ca_cert, cert, private_key} and/or {ca_cert_src, cert_src, private_key_src}.

    • If the {ca_cert_src, cert_src, private_key_src} triplet is set, the default locations /etc/pki/tls/certs and /etc/pki/tls/private are used as the destination on the managed node to transfer files from control node. In this case, the file names are identical to the original ones in the triplet
    • If the {ca_cert, cert, private_key} triplet is set, files are expected to be on the default path before the logging configuration.
    • If both triplets are set, files are transferred from local path from control node to specific path of the managed node.
    ca_cert
    Represents the path to CA certificate. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    cert
    Represents the path to certificate. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    private_key
    Represents the path to private key. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    ca_cert_src
    Represents local CA certificate file path which is copied to the target host. If ca_cert is specified, it is copied to the location.
    cert_src
    Represents the local certificate file path which is copied to the target host. If cert is specified, it is copied to the location.
    private_key_src
    Represents the local key file path which is copied to the target host. If private_key is specified, it is copied to the location.
    pki_authmode
    Accepts the authentication mode as name or fingerprint.
    permitted_servers
    List of servers that will be allowed by the logging client to connect and send logs over TLS.
    inputs
    List of logging input dictionary.
    outputs
    List of logging output dictionary.
  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.logging/README.md file
  • /usr/share/doc/rhel-system-roles/logging/ directory

9.2.6.2. Configuring server logging with RELP

You can use the logging RHEL system role to configure logging in RHEL systems as a server and can receive logs from the remote logging system with RELP by running an Ansible playbook.

This procedure configures RELP on all hosts in the server group in the Ansible inventory. The RELP configuration uses TLS to encrypt the message transmission for secure transfer of logs over the network.

Prerequisites

Procedure

  1. Create a playbook file, for example ~/playbook.yml, with the following content:

    ---
    - name: Deploying remote input and remote_files output
      hosts: managed-node-01.example.com
      roles:
        - rhel-system-roles.logging
      vars:
        logging_inputs:
          - name: relp_server
            type: relp
            port: 20514
            tls: true
            ca_cert: /etc/pki/tls/certs/ca.pem
            cert: /etc/pki/tls/certs/server-cert.pem
            private_key: /etc/pki/tls/private/server-key.pem
            pki_authmode: name
            permitted_clients:
              - '*example.client.com'
        logging_outputs:
          - name: remote_files_output
            type: remote_files
        logging_flows:
          - name: example_flow
            inputs: relp_server
            outputs: remote_files_output

    The playbooks uses the following settings:

    port
    Port number the remote logging system is listening.
    tls

    Ensures secure transfer of logs over the network. If you do not want a secure wrapper you can set the tls variable to false. By default tls parameter is set to true while working with RELP and requires key/certificates and triplets {ca_cert, cert, private_key} and/or {ca_cert_src, cert_src, private_key_src}.

    • If the {ca_cert_src, cert_src, private_key_src} triplet is set, the default locations /etc/pki/tls/certs and /etc/pki/tls/private are used as the destination on the managed node to transfer files from control node. In this case, the file names are identical to the original ones in the triplet
    • If the {ca_cert, cert, private_key} triplet is set, files are expected to be on the default path before the logging configuration.
    • If both triplets are set, files are transferred from local path from control node to specific path of the managed node.
    ca_cert
    Represents the path to CA certificate. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    cert
    Represents the path to the certificate. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    private_key
    Represents the path to private key. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    ca_cert_src
    Represents local CA certificate file path which is copied to the target host. If ca_cert is specified, it is copied to the location.
    cert_src
    Represents the local certificate file path which is copied to the target host. If cert is specified, it is copied to the location.
    private_key_src
    Represents the local key file path which is copied to the target host. If private_key is specified, it is copied to the location.
    pki_authmode
    Accepts the authentication mode as name or fingerprint.
    permitted_clients
    List of clients that will be allowed by the logging server to connect and send logs over TLS.
    inputs
    List of logging input dictionary.
    outputs
    List of logging output dictionary.
  2. Validate the playbook syntax:

    $ ansible-playbook --syntax-check ~/playbook.yml

    Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

  3. Run the playbook:

    $ ansible-playbook ~/playbook.yml

Additional resources

  • /usr/share/ansible/roles/rhel-system-roles.logging/README.md file
  • /usr/share/doc/rhel-system-roles/logging/ directory

9.2.7. Additional resources

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.