Chapter 14. Configuring a remote logging solution

Procedure

1. Optional: To use a different port for rsyslog traffic, add the syslogd_port_t SELinux type to port. For example, enable port 30514:

   # semanage port -a -t syslogd_port_t -p tcp 30514

2. Optional: To use a different port for rsyslog traffic, configure firewalld to allow incoming rsyslog traffic on that port. For example, allow TCP traffic on port 30514:

   # firewall-cmd --zone=<zone-name> --permanent --add-port=30514/tcp
   success
   # firewall-cmd --reload

3. Create a new file in the /etc/rsyslog.d/ directory named, for example, remotelog.conf, and insert the following content:

   # Define templates before the rules that use them
   # Per-Host templates for remote systems
   template(name="TmplAuthpriv" type="list") {
       constant(value="/var/log/remote/auth/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   template(name="TmplMsg" type="list") {
       constant(value="/var/log/remote/msg/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   # Provides TCP syslog reception
   module(load="imtcp")
   
   # Adding this ruleset to process remote messages
   ruleset(name="remote1"){
        authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
         *.info;mail.none;authpriv.none;cron.none
   action(type="omfile" DynaFile="TmplMsg")
   }
   
   input(type="imtcp" port="30514" ruleset="remote1")

4. Save the changes to the /etc/rsyslog.d/remotelog.conf file.

5. Test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-2.el8, config validation run...
   rsyslogd: End of config validation run. Bye.

6. Make sure the rsyslog service is running and enabled on the logging server:

   # systemctl status rsyslog

7. Restart the rsyslog service.

   # systemctl restart rsyslog

8. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

Procedure

1. Create a new file in the /etc/rsyslog.d/ directory named, for example, 10-remotelog.conf, and insert the following content:

   *.* action(type="omfwd"
         queue.type="linkedlist"
         queue.filename="example_fwd"
         action.resumeRetryCount="-1"
         queue.saveOnShutdown="on"
         target="example.com" port="30514" protocol="tcp"
        )

   Where:

   • The queue.type="linkedlist" setting enables a LinkedList in-memory queue,
   • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
   • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
   • The queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.

   • The last line forwards all received messages to the logging server. Port specification is optional.

     With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

   Note

   Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

2. Restart the rsyslog service.

   # systemctl restart rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the /var/log/messages log, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Configure the server for receiving encrypted logs from your client systems:

   1. Create a new file in the /etc/rsyslog.d/ directory named, for example, securelogser.conf.

   2. To encrypt the communication, the configuration file must contain paths to certificate files on your server, a selected authentication method, and a stream driver that supports TLS encryption. Add the following lines to the /etc/rsyslog.d/securelogser.conf file:

      # Set certificate files
      global(
        DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
        DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/server-cert.pem"
        DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/server-key.pem"
      )
      
      # TCP listener
      module(
        load="imtcp"
        PermittedPeer=["client1.example.com", "client2.example.com"]
        StreamDriver.AuthMode="x509/name"
        StreamDriver.Mode="1"
        StreamDriver.Name="ossl"
      )
      
      # Start up listener at port 514
      input(
        type="imtcp"
        port="514"
      )

      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option. See the documentation installed with the rsyslog-doc package for more information about less strict authentication modes than x509/name.

   3. Optional: From Rsyslog version 8.2310, which is provided in RHEL 9.4, you can customize the connection configuration. To do so, replace the input section with the following:

      input(
        type="imtcp"
        Port="50515"
        StreamDriver.Name="<driver>"
        streamdriver.CAFile="/etc/rsyslog.d/<ca1>.pem"
        streamdriver.CertFile="/etc/rsyslog.d/<server1-cert>.pem"
        streamdriver.KeyFile="/etc/rsyslog.d/<server1-key>.pem"
      )

      • Replace <driver> with ossl or gtls depending on the driver you want to use.
      • Replace <ca1> with the CA certificate, <server1-cert> with the certificate, and <server1-key> with the key of the customized connection.
   4. Save the changes to the /etc/rsyslog.d/securelogser.conf file.

   5. Verify the syntax of the /etc/rsyslog.conf file and any files in the /etc/rsyslog.d/ directory:

      # rsyslogd -N 1
      rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
      rsyslogd: End of config validation run. Bye.

   6. Make sure the rsyslog service is running and enabled on the logging server:

      # systemctl status rsyslog

   7. Restart the rsyslog service:

      # systemctl restart rsyslog

   8. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

2. Configure clients for sending encrypted logs to the server:

   1. On a client system, create a new file in the /etc/rsyslog.d/ directory named, for example, securelogcli.conf.

   2. Add the following lines to the /etc/rsyslog.d/securelogcli.conf file:

      # Set certificate files
      global(
        DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
        DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/client-cert.pem"
        DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/client-key.pem"
      )
      
      
      # Set up the action for all messages
      *.* action(
        type="omfwd"
        StreamDriver="ossl"
        StreamDriverMode="1"
        StreamDriverPermittedPeers="server.example.com"
        StreamDriverAuthMode="x509/name"
        target="server.example.com" port="514" protocol="tcp"
      )

      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option.

   3. Optional: From Rsyslog version 8.2310, which is provided in RHEL 9.4, you can customize the connection configuration. To do so, replace the action section with the following:

      local1.* action(
        type="omfwd"
        StreamDriver="<driver>"
        StreamDriverMode="1"
        StreamDriverAuthMode="x509/certvalid"
        streamDriver.CAFile="/etc/rsyslog.d/<ca1>.pem"
        streamDriver.CertFile="/etc/rsyslog.d/<client1-cert>.pem"
        streamDriver.KeyFile="/etc/rsyslog.d/<client1-key>.pem"
        target="server.example.com" port="514" protocol="tcp"
        )

      • Replace <driver> with ossl or gtls depending on the driver you want to use.
      • Replace <ca1> with the CA certificate, <client1-cert> with the certificate, and <client1-key> with the key of the customized connection.
   4. Save the changes to the /etc/rsyslog.d/securelogcli.conf file.

   5. Verify the syntax of the /etc/rsyslog.conf file and other files in the /etc/rsyslog.d/ directory:

      # rsyslogd -N 1
      rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
      rsyslogd: End of config validation run. Bye.

   6. Make sure the rsyslog service is running and enabled on the logging server:

      # systemctl status rsyslog

   7. Restart the rsyslog service:

      # systemctl restart rsyslog

   8. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the /var/log/messages log, for example:

   # cat /var/log/remote/msg/<hostname>/root.log
   Feb 25 03:53:17 <hostname> root[6064]: test

   Where <hostname> is the hostname of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Optional: To use a different port for rsyslog traffic than the default port 514:

   1. Add the syslogd_port_t SELinux type to the SELinux policy configuration, replacing portno with the port number you want rsyslog to use:

      # semanage port -a -t syslogd_port_t -p udp portno

   2. Configure firewalld to allow incoming rsyslog traffic, replacing portno with the port number and zone with the zone you want rsyslog to use:

      # firewall-cmd --zone=zone --permanent --add-port=portno/udp
      success
      # firewall-cmd --reload

   3. Reload the firewall rules:

      # firewall-cmd --reload

2. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, remotelogserv.conf, and insert the following content:

   # Define templates before the rules that use them
   # Per-Host templates for remote systems
   template(name="TmplAuthpriv" type="list") {
       constant(value="/var/log/remote/auth/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   template(name="TmplMsg" type="list") {
       constant(value="/var/log/remote/msg/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   # Provides UDP syslog reception
   module(load="imudp")
   
   # This ruleset processes remote messages
   ruleset(name="remote1"){
        authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
         *.info;mail.none;authpriv.none;cron.none
   action(type="omfile" DynaFile="TmplMsg")
   }
   
   input(type="imudp" port="514" ruleset="remote1")

   Where 514 is the port number rsyslog uses by default. You can specify a different port instead.

3. Verify the syntax of the /etc/rsyslog.conf file and all .conf files in the /etc/rsyslog.d/ directory:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-2.el8, config validation run...

4. Restart the rsyslog service.

   # systemctl restart rsyslog

5. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

Procedure

1. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, 10-remotelogcli.conf, and insert the following content:

   *.* action(type="omfwd"
         queue.type="linkedlist"
         queue.filename="example_fwd"
         action.resumeRetryCount="-1"
         queue.saveOnShutdown="on"
         target="example.com" port="portno" protocol="udp"
        )

   Where:

   • The queue.type="linkedlist" setting enables a LinkedList in-memory queue.
   • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
   • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if the server is not responding.
   • The enabled queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.
   • The portno value is the port number you want rsyslog to use. The default value is 514.

   • The last line forwards all received messages to the logging server, port specification is optional.

     With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

   Note

   Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

2. Restart the rsyslog service.

   # systemctl restart rsyslog

3. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the /var/log/remote/msg/hostname/root.log log, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Configure the client system for reliable remote logging:

   1. On the client system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpclient.conf, and insert the following content:

      module(load="omrelp")
      *.* action(type="omrelp" target="_target_IP_" port="_target_port_")

      Where:

      • target_IP is the IP address of the logging server.
      • target_port is the port of the logging server.
   2. Save the changes to the /etc/rsyslog.d/relpclient.conf file.

   3. Restart the rsyslog service.

      # systemctl restart rsyslog

   4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

2. Configure the server system for reliable remote logging:

   1. On the server system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpserv.conf, and insert the following content:

      ruleset(name="relp"){
      *.* action(type="omfile" file="_log_path_")
      }
      
      
      module(load="imrelp")
      input(type="imrelp" port="_target_port_" ruleset="relp")

      Where:

      • log_path specifies the path for storing messages.
      • target_port is the port of the logging server. Use the same value as in the client configuration file.
   2. Save the changes to the /etc/rsyslog.d/relpserv.conf file.

   3. Restart the rsyslog service.

      # systemctl restart rsyslog

   4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the log at the specified log_path, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Install the netconsole-service package:

   # dnf install netconsole-service

2. Edit the /etc/sysconfig/netconsole file and set the SYSLOGADDR parameter to the IP address of the remote host:

   # SYSLOGADDR=192.0.2.1

3. Enable and start the netconsole service:

   # systemctl enable --now netconsole