Chapter 9. Configuring logging

Procedure

1. Optional: To use a different port for rsyslog traffic, add the syslogd_port_t SELinux type to port. For example, enable port 30514:

   # semanage port -a -t syslogd_port_t -p tcp 30514

2. Optional: To use a different port for rsyslog traffic, configure firewalld to allow incoming rsyslog traffic on that port. For example, allow TCP traffic on port 30514:

   # firewall-cmd --zone=<zone-name> --permanent --add-port=30514/tcp
   success
   # firewall-cmd --reload

3. Create a new file in the /etc/rsyslog.d/ directory named, for example, remotelog.conf, and insert the following content:

   # Define templates before the rules that use them
   # Per-Host templates for remote systems
   template(name="TmplAuthpriv" type="list") {
       constant(value="/var/log/remote/auth/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   template(name="TmplMsg" type="list") {
       constant(value="/var/log/remote/msg/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   # Provides TCP syslog reception
   module(load="imtcp")
   
   # Adding this ruleset to process remote messages
   ruleset(name="remote1"){
        authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
         *.info;mail.none;authpriv.none;cron.none
   action(type="omfile" DynaFile="TmplMsg")
   }
   
   input(type="imtcp" port="30514" ruleset="remote1")

4. Save the changes to the /etc/rsyslog.d/remotelog.conf file.

5. Test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-2.el8, config validation run...
   rsyslogd: End of config validation run. Bye.

6. Make sure the rsyslog service is running and enabled on the logging server:

   # systemctl status rsyslog

7. Restart the rsyslog service.

   # systemctl restart rsyslog

8. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

Procedure

1. Create a new file in the /etc/rsyslog.d/ directory named, for example, 10-remotelog.conf, and insert the following content:

   *.* action(type="omfwd"
         queue.type="linkedlist"
         queue.filename="example_fwd"
         action.resumeRetryCount="-1"
         queue.saveOnShutdown="on"
         target="example.com" port="30514" protocol="tcp"
        )

   Where:

   • The queue.type="linkedlist" setting enables a LinkedList in-memory queue,
   • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
   • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
   • The queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.

   • The last line forwards all received messages to the logging server. Port specification is optional.

     With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

   Note

   Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

2. Restart the rsyslog service.

   # systemctl restart rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the /var/log/messages log, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Configure the server for receiving encrypted logs from your client systems:

   1. Create a new file in the /etc/rsyslog.d/ directory named, for example, securelogser.conf.

   2. To encrypt the communication, the configuration file must contain paths to certificate files on your server, a selected authentication method, and a stream driver that supports TLS encryption. Add the following lines to the /etc/rsyslog.d/securelogser.conf file:

      # Set certificate files
      global(
        DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
        DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/server-cert.pem"
        DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/server-key.pem"
      )
      
      # TCP listener
      module(
        load="imtcp"
        PermittedPeer=["client1.example.com", "client2.example.com"]
        StreamDriver.AuthMode="x509/name"
        StreamDriver.Mode="1"
        StreamDriver.Name="ossl"
      )
      
      # Start up listener at port 514
      input(
        type="imtcp"
        port="514"
      )

      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option. See the documentation installed with the rsyslog-doc package for more information about less strict authentication modes than x509/name.

   3. Save the changes to the /etc/rsyslog.d/securelogser.conf file.

   4. Verify the syntax of the /etc/rsyslog.conf file and any files in the /etc/rsyslog.d/ directory:

      # rsyslogd -N 1
      rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
      rsyslogd: End of config validation run. Bye.

   5. Make sure the rsyslog service is running and enabled on the logging server:

      # systemctl status rsyslog

   6. Restart the rsyslog service:

      # systemctl restart rsyslog

   7. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

2. Configure clients for sending encrypted logs to the server:

   1. On a client system, create a new file in the /etc/rsyslog.d/ directory named, for example, securelogcli.conf.

   2. Add the following lines to the /etc/rsyslog.d/securelogcli.conf file:

      # Set certificate files
      global(
        DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
        DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/client-cert.pem"
        DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/client-key.pem"
      )
      
      
      # Set up the action for all messages
      *.* action(
        type="omfwd"
        StreamDriver="ossl"
        StreamDriverMode="1"
        StreamDriverPermittedPeers="server.example.com"
        StreamDriverAuthMode="x509/name"
        target="server.example.com" port="514" protocol="tcp"
      )

      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option.

   3. Save the changes to the /etc/rsyslog.d/securelogcli.conf file.

   4. Verify the syntax of the /etc/rsyslog.conf file and other files in the /etc/rsyslog.d/ directory:

      # rsyslogd -N 1
      rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
      rsyslogd: End of config validation run. Bye.

   5. Make sure the rsyslog service is running and enabled on the logging server:

      # systemctl status rsyslog

   6. Restart the rsyslog service:

      # systemctl restart rsyslog

   7. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the /var/log/messages log, for example:

   # cat /var/log/remote/msg/<hostname>/root.log
   Feb 25 03:53:17 <hostname> root[6064]: test

   Where <hostname> is the hostname of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Optional: To use a different port for rsyslog traffic than the default port 514:

   1. Add the syslogd_port_t SELinux type to the SELinux policy configuration, replacing portno with the port number you want rsyslog to use:

      # semanage port -a -t syslogd_port_t -p udp portno

   2. Configure firewalld to allow incoming rsyslog traffic, replacing portno with the port number and zone with the zone you want rsyslog to use:

      # firewall-cmd --zone=zone --permanent --add-port=portno/udp
      success
      # firewall-cmd --reload

   3. Reload the firewall rules:

      # firewall-cmd --reload

2. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, remotelogserv.conf, and insert the following content:

   # Define templates before the rules that use them
   # Per-Host templates for remote systems
   template(name="TmplAuthpriv" type="list") {
       constant(value="/var/log/remote/auth/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   template(name="TmplMsg" type="list") {
       constant(value="/var/log/remote/msg/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   # Provides UDP syslog reception
   module(load="imudp")
   
   # This ruleset processes remote messages
   ruleset(name="remote1"){
        authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
         *.info;mail.none;authpriv.none;cron.none
   action(type="omfile" DynaFile="TmplMsg")
   }
   
   input(type="imudp" port="514" ruleset="remote1")

   Where 514 is the port number rsyslog uses by default. You can specify a different port instead.

3. Verify the syntax of the /etc/rsyslog.conf file and all .conf files in the /etc/rsyslog.d/ directory:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-2.el8, config validation run...

4. Restart the rsyslog service.

   # systemctl restart rsyslog

5. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

Procedure

1. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, 10-remotelogcli.conf, and insert the following content:

   *.* action(type="omfwd"
         queue.type="linkedlist"
         queue.filename="example_fwd"
         action.resumeRetryCount="-1"
         queue.saveOnShutdown="on"
         target="example.com" port="portno" protocol="udp"
        )

   Where:

   • The queue.type="linkedlist" setting enables a LinkedList in-memory queue.
   • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
   • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if the server is not responding.
   • The enabled queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.
   • The portno value is the port number you want rsyslog to use. The default value is 514.

   • The last line forwards all received messages to the logging server, port specification is optional.

     With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

   Note

   Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

2. Restart the rsyslog service.

   # systemctl restart rsyslog

3. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the /var/log/remote/msg/hostname/root.log log, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Configure the client system for reliable remote logging:

   1. On the client system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpclient.conf, and insert the following content:

      module(load="omrelp")
      *.* action(type="omrelp" target="_target_IP_" port="_target_port_")

      Where:

      • target_IP is the IP address of the logging server.
      • target_port is the port of the logging server.
   2. Save the changes to the /etc/rsyslog.d/relpclient.conf file.

   3. Restart the rsyslog service.

      # systemctl restart rsyslog

   4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

2. Configure the server system for reliable remote logging:

   1. On the server system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpserv.conf, and insert the following content:

      ruleset(name="relp"){
      *.* action(type="omfile" file="_log_path_")
      }
      
      
      module(load="imrelp")
      input(type="imrelp" port="_target_port_" ruleset="relp")

      Where:

      • log_path specifies the path for storing messages.
      • target_port is the port of the logging server. Use the same value as in the client configuration file.
   2. Save the changes to the /etc/rsyslog.d/relpserv.conf file.

   3. Restart the rsyslog service.

      # systemctl restart rsyslog

   4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the log at the specified log_path, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Install the netconsole-service package:

   # yum install netconsole-service

2. Edit the /etc/sysconfig/netconsole file and set the SYSLOGADDR parameter to the IP address of the remote host:

   # SYSLOGADDR=192.0.2.1

3. Enable and start the netconsole service:

   # systemctl enable --now netconsole

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Deploying basics input and implicit files output
     hosts: managed-node-01.example.com
     roles:
       - rhel-system-roles.logging
     vars:
       logging_inputs:
         - name: system_input
           type: basics
       logging_outputs:
         - name: files_output
           type: files
       logging_flows:
         - name: flow1
           inputs: [system_input]
           outputs: [files_output]

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Verification

1. Test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-6.el8, config validation run...
   rsyslogd: End of config validation run. Bye.

2. Verify that the system sends messages to the log:

   1. Send a test message:

      # logger test

   2. View the /var/log/messages log, for example:

      # cat /var/log/messages
      Aug  5 13:48:31 <hostname> root[6778]: test

      Where <hostname> is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Deploying files input and configured files output
     hosts: managed-node-01.example.com
     roles:
       - rhel-system-roles.logging
     vars:
       logging_inputs:
         - name: files_input
           type: basics
       logging_outputs:
         - name: files_output0
           type: files
           property: msg
           property_op: contains
           property_value: error
           path: /var/log/errors.log
         - name: files_output1
           type: files
           property: msg
           property_op: "!contains"
           property_value: error
           path: /var/log/others.log
       logging_flows:
         - name: flow0
           inputs: [files_input]
           outputs: [files_output0, files_output1]

   Using this configuration, all messages that contain the error string are logged in /var/log/errors.log, and all other messages are logged in /var/log/others.log.

   You can replace the error property value with the string by which you want to filter.

   You can modify the variables according to your preferences.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Verification

1. Test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-6.el8, config validation run...
   rsyslogd: End of config validation run. Bye.

2. Verify that the system sends messages that contain the error string to the log:

   1. Send a test message:

      # logger error

   2. View the /var/log/errors.log log, for example:

      # cat /var/log/errors.log
      Aug  5 13:48:31 hostname root[6778]: error

      Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Deploying remote input and remote_files output
     hosts: managed-node-01.example.com
     roles:
       - rhel-system-roles.logging
     vars:
       logging_inputs:
         - name: remote_udp_input
           type: remote
           udp_ports: [ 601 ]
         - name: remote_tcp_input
           type: remote
           tcp_ports: [ 601 ]
       logging_outputs:
         - name: remote_files_output
           type: remote_files
       logging_flows:
         - name: flow_0
           inputs: [remote_udp_input, remote_tcp_input]
           outputs: [remote_files_output]
   
   - name: Deploying basics input and forwards output
     hosts: managed-node-02.example.com
     roles:
       - rhel-system-roles.logging
     vars:
       logging_inputs:
         - name: basic_input
           type: basics
       logging_outputs:
         - name: forward_output0
           type: forwards
           severity: info
           target: <host1.example.com>
           udp_port: 601
         - name: forward_output1
           type: forwards
           facility: mail
           target: <host1.example.com>
           tcp_port: 601
       logging_flows:
         - name: flows0
           inputs: [basic_input]
           outputs: [forward_output0, forward_output1]
   
   [basic_input]
   [forward_output0, forward_output1]

   Where <host1.example.com> is the logging server.

   Note

   You can modify the parameters in the playbook to fit your needs.

   Warning

   The logging solution works only with the ports defined in the SELinux policy of the server or client system and open in the firewall. The default SELinux policy includes ports 601, 514, 6514, 10514, and 20514. To use a different port, modify the SELinux policy on the client and server systems.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Verification

1. On both the client and the server system, test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-6.el8, config validation run (level 1), master config /etc/rsyslog.conf
   rsyslogd: End of config validation run. Bye.

2. Verify that the client system sends messages to the server:

   1. On the client system, send a test message:

      # logger test

   2. On the server system, view the /var/log/<host2.example.com>/messages log, for example:

      # cat /var/log/<host2.example.com>/messages
      Aug  5 13:48:31 <host2.example.com> root[6778]: test

      Where <host2.example.com> is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.