Chapter 8. Configuring logging

Procedure

1. Optional: To use a different port for rsyslog traffic, add the syslogd_port_t SELinux type to port. For example, enable port 30514:

   # semanage port -a -t syslogd_port_t -p tcp 30514

2. Optional: To use a different port for rsyslog traffic, configure firewalld to allow incoming rsyslog traffic on that port. For example, allow TCP traffic on port 30514:

   # firewall-cmd --zone=<zone-name> --permanent --add-port=30514/tcp
   success
   # firewall-cmd --reload

3. Create a new file in the /etc/rsyslog.d/ directory named, for example, remotelog.conf, and insert the following content:

   # Define templates before the rules that use them
   # Per-Host templates for remote systems
   template(name="TmplAuthpriv" type="list") {
       constant(value="/var/log/remote/auth/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   template(name="TmplMsg" type="list") {
       constant(value="/var/log/remote/msg/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   # Provides TCP syslog reception
   module(load="imtcp")
   
   # Adding this ruleset to process remote messages
   ruleset(name="remote1"){
        authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
         *.info;mail.none;authpriv.none;cron.none
   action(type="omfile" DynaFile="TmplMsg")
   }
   
   input(type="imtcp" port="30514" ruleset="remote1")

4. Save the changes to the /etc/rsyslog.d/remotelog.conf file.

5. Test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-2.el8, config validation run...
   rsyslogd: End of config validation run. Bye.

6. Make sure the rsyslog service is running and enabled on the logging server:

   # systemctl status rsyslog

7. Restart the rsyslog service.

   # systemctl restart rsyslog

8. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

Procedure

1. Create a new file in the /etc/rsyslog.d/ directory named, for example, 10-remotelog.conf, and insert the following content:

   *.* action(type="omfwd"
         queue.type="linkedlist"
         queue.filename="example_fwd"
         action.resumeRetryCount="-1"
         queue.saveOnShutdown="on"
         target="example.com" port="30514" protocol="tcp"
        )

   Where:

   • The queue.type="linkedlist" setting enables a LinkedList in-memory queue,
   • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
   • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
   • The queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.

   • The last line forwards all received messages to the logging server. Port specification is optional.

     With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

   Note

   Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

2. Restart the rsyslog service.

   # systemctl restart rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the /var/log/messages log, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Configure the server for receiving encrypted logs from your client systems:

   1. Create a new file in the /etc/rsyslog.d/ directory named, for example, securelogser.conf.

   2. To encrypt the communication, the configuration file must contain paths to certificate files on your server, a selected authentication method, and a stream driver that supports TLS encryption. Add the following lines to the /etc/rsyslog.d/securelogser.conf file:

      # Set certificate files
      global(
        DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
        DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/server-cert.pem"
        DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/server-key.pem"
      )
      
      # TCP listener
      module(
        load="imtcp"
        PermittedPeer=["client1.example.com", "client2.example.com"]
        StreamDriver.AuthMode="x509/name"
        StreamDriver.Mode="1"
        StreamDriver.Name="ossl"
      )
      
      # Start up listener at port 514
      input(
        type="imtcp"
        port="514"
      )

      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option. See the documentation installed with the rsyslog-doc package for more information about less strict authentication modes than x509/name.

   3. Save the changes to the /etc/rsyslog.d/securelogser.conf file.

   4. Verify the syntax of the /etc/rsyslog.conf file and any files in the /etc/rsyslog.d/ directory:

      # rsyslogd -N 1
      rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
      rsyslogd: End of config validation run. Bye.

   5. Make sure the rsyslog service is running and enabled on the logging server:

      # systemctl status rsyslog

   6. Restart the rsyslog service:

      # systemctl restart rsyslog

   7. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

2. Configure clients for sending encrypted logs to the server:

   1. On a client system, create a new file in the /etc/rsyslog.d/ directory named, for example, securelogcli.conf.

   2. Add the following lines to the /etc/rsyslog.d/securelogcli.conf file:

      # Set certificate files
      global(
        DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
        DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/client-cert.pem"
        DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/client-key.pem"
      )
      
      
      # Set up the action for all messages
      *.* action(
        type="omfwd"
        StreamDriver="ossl"
        StreamDriverMode="1"
        StreamDriverPermittedPeers="server.example.com"
        StreamDriverAuthMode="x509/name"
        target="server.example.com" port="514" protocol="tcp"
      )

      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option.

   3. Save the changes to the /etc/rsyslog.d/securelogcli.conf file.

   4. Verify the syntax of the /etc/rsyslog.conf file and other files in the /etc/rsyslog.d/ directory:

      # rsyslogd -N 1
      rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
      rsyslogd: End of config validation run. Bye.

   5. Make sure the rsyslog service is running and enabled on the logging server:

      # systemctl status rsyslog

   6. Restart the rsyslog service:

      # systemctl restart rsyslog

   7. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the /var/log/messages log, for example:

   # cat /var/log/remote/msg/<hostname>/root.log
   Feb 25 03:53:17 <hostname> root[6064]: test

   Where <hostname> is the hostname of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Optional: To use a different port for rsyslog traffic than the default port 514:

   1. Add the syslogd_port_t SELinux type to the SELinux policy configuration, replacing portno with the port number you want rsyslog to use:

      # semanage port -a -t syslogd_port_t -p udp portno

   2. Configure firewalld to allow incoming rsyslog traffic, replacing portno with the port number and zone with the zone you want rsyslog to use:

      # firewall-cmd --zone=zone --permanent --add-port=portno/udp
      success
      # firewall-cmd --reload

   3. Reload the firewall rules:

      # firewall-cmd --reload

2. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, remotelogserv.conf, and insert the following content:

   # Define templates before the rules that use them
   # Per-Host templates for remote systems
   template(name="TmplAuthpriv" type="list") {
       constant(value="/var/log/remote/auth/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   template(name="TmplMsg" type="list") {
       constant(value="/var/log/remote/msg/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   # Provides UDP syslog reception
   module(load="imudp")
   
   # This ruleset processes remote messages
   ruleset(name="remote1"){
        authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
         *.info;mail.none;authpriv.none;cron.none
   action(type="omfile" DynaFile="TmplMsg")
   }
   
   input(type="imudp" port="514" ruleset="remote1")

   Where 514 is the port number rsyslog uses by default. You can specify a different port instead.

3. Verify the syntax of the /etc/rsyslog.conf file and all .conf files in the /etc/rsyslog.d/ directory:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-2.el8, config validation run...

4. Restart the rsyslog service.

   # systemctl restart rsyslog

5. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

Procedure

1. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, 10-remotelogcli.conf, and insert the following content:

   *.* action(type="omfwd"
         queue.type="linkedlist"
         queue.filename="example_fwd"
         action.resumeRetryCount="-1"
         queue.saveOnShutdown="on"
         target="example.com" port="portno" protocol="udp"
        )

   Where:

   • The queue.type="linkedlist" setting enables a LinkedList in-memory queue.
   • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
   • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if the server is not responding.
   • The enabled queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.
   • The portno value is the port number you want rsyslog to use. The default value is 514.

   • The last line forwards all received messages to the logging server, port specification is optional.

     With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

   Note

   Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

2. Restart the rsyslog service.

   # systemctl restart rsyslog

3. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the /var/log/remote/msg/hostname/root.log log, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Configure the client system for reliable remote logging:

   1. On the client system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpclient.conf, and insert the following content:

      module(load="omrelp")
      *.* action(type="omrelp" target="_target_IP_" port="_target_port_")

      Where:

      • target_IP is the IP address of the logging server.
      • target_port is the port of the logging server.
   2. Save the changes to the /etc/rsyslog.d/relpclient.conf file.

   3. Restart the rsyslog service.

      # systemctl restart rsyslog

   4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

2. Configure the server system for reliable remote logging:

   1. On the server system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpserv.conf, and insert the following content:

      ruleset(name="relp"){
      *.* action(type="omfile" file="_log_path_")
      }
      
      
      module(load="imrelp")
      input(type="imrelp" port="_target_port_" ruleset="relp")

      Where:

      • log_path specifies the path for storing messages.
      • target_port is the port of the logging server. Use the same value as in the client configuration file.
   2. Save the changes to the /etc/rsyslog.d/relpserv.conf file.

   3. Restart the rsyslog service.

      # systemctl restart rsyslog

   4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

1. On the client system, send a test message:

   # logger test

2. On the server system, view the log at the specified log_path, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Install the netconsole-service package:

   # yum install netconsole-service

2. Edit the /etc/sysconfig/netconsole file and set the SYSLOGADDR parameter to the IP address of the remote host:

   # SYSLOGADDR=192.0.2.1

3. Enable and start the netconsole service:

   # systemctl enable --now netconsole

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Deploy the logging solution
     hosts: managed-node-01.example.com
     tasks:
       - name: Filter logs based on a specific value they contain
         ansible.builtin.include_role:
           name: rhel-system-roles.logging
         vars:
           logging_inputs:
             - name: files_input
               type: basics
           logging_outputs:
             - name: files_output0
               type: files
               property: msg
               property_op: contains
               property_value: error
               path: /var/log/errors.log
             - name: files_output1
               type: files
               property: msg
               property_op: "!contains"
               property_value: error
               path: /var/log/others.log
           logging_flows:
             - name: flow0
               inputs: [files_input]
               outputs: [files_output0, files_output1]

   The settings specified in the example playbook include the following:

   logging_inputs

   Defines a list of logging input dictionaries. The type: basics option covers inputs from systemd journal or Unix socket.

   logging_outputs

   Defines a list of logging output dictionaries. The type: files option supports storing logs in the local files, usually in the /var/log/ directory. The property: msg; property: contains; and property_value: error options specify that all logs that contain the error string are stored in the /var/log/errors.log file. The property: msg; property: !contains; and property_value: error options specify that all other logs are put in the /var/log/others.log file. You can replace the error value with the string by which you want to filter.

   logging_flows

   Defines a list of logging flow dictionaries to specify relationships between logging_inputs and logging_outputs. The inputs: [files_input] option specifies a list of inputs, from which processing of logs starts. The outputs: [files_output0, files_output1] option specifies a list of outputs, to which the logs are sent.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.logging/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Verification

1. On the managed node, test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-6.el8, config validation run...
   rsyslogd: End of config validation run. Bye.

2. On the managed node, verify that the system sends messages that contain the error string to the log:

   1. Send a test message:

      # logger error

   2. View the /var/log/errors.log log, for example:

      # cat /var/log/errors.log
      Aug  5 13:48:31 hostname root[6778]: error

      Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Deploy the logging solution
     hosts: managed-node-01.example.com
     tasks:
       - name: Configure the server to receive remote input
         ansible.builtin.include_role:
           name: rhel-system-roles.logging
         vars:
           logging_inputs:
             - name: remote_udp_input
               type: remote
               udp_ports: [ 601 ]
             - name: remote_tcp_input
               type: remote
               tcp_ports: [ 601 ]
           logging_outputs:
             - name: remote_files_output
               type: remote_files
           logging_flows:
             - name: flow_0
               inputs: [remote_udp_input, remote_tcp_input]
               outputs: [remote_files_output]
   
   - name: Deploy the logging solution
     hosts: managed-node-02.example.com
     tasks:
       - name: Configure the server to output the logs to local files in directories named by remote host names
         ansible.builtin.include_role:
           name: rhel-system-roles.logging
         vars:
           logging_inputs:
             - name: basic_input
               type: basics
           logging_outputs:
             - name: forward_output0
               type: forwards
               severity: info
               target: <host1.example.com>
               udp_port: 601
             - name: forward_output1
               type: forwards
               facility: mail
               target: <host1.example.com>
               tcp_port: 601
           logging_flows:
             - name: flows0
               inputs: [basic_input]
               outputs: [forward_output0, forward_output1]
   
   [basic_input]
   [forward_output0, forward_output1]

   The settings specified in the first play of the example playbook include the following:

   logging_inputs

   Defines a list of logging input dictionaries. The type: remote option covers remote inputs from the other logging system over the network. The udp_ports: [ 601 ] option defines a list of UDP port numbers to monitor. The tcp_ports: [ 601 ] option defines a list of TCP port numbers to monitor. If both udp_ports and tcp_ports is set, udp_ports is used and tcp_ports is dropped.

   logging_outputs

   Defines a list of logging output dictionaries. The type: remote_files option makes output store logs to the local files per remote host and program name originated the logs.

   logging_flows

   Defines a list of logging flow dictionaries to specify relationships between logging_inputs and logging_outputs. The inputs: [remote_udp_input, remote_tcp_input] option specifies a list of inputs, from which processing of logs starts. The outputs: [remote_files_output] option specifies a list of outputs, to which the logs are sent.

   The settings specified in the second play of the example playbook include the following:

   logging_inputs

   Defines a list of logging input dictionaries. The type: basics option covers inputs from systemd journal or Unix socket.

   logging_outputs

   Defines a list of logging output dictionaries. The type: forwards option supports sending logs to the remote logging server over the network. The severity: info option refers to log messages of the informative importance. The facility: mail option refers to the type of system program that is generating the log message. The target: <host1.example.com> option specifies the hostname of the remote logging server. The udp_port: 601/tcp_port: 601 options define the UDP/TCP ports on which the remote logging server listens.

   logging_flows

   Defines a list of logging flow dictionaries to specify relationships between logging_inputs and logging_outputs. The inputs: [basic_input] option specifies a list of inputs, from which processing of logs starts. The outputs: [forward_output0, forward_output1] option specifies a list of outputs, to which the logs are sent.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.logging/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Verification

1. On both the client and the server system, test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-6.el8, config validation run (level 1), master config /etc/rsyslog.conf
   rsyslogd: End of config validation run. Bye.

2. Verify that the client system sends messages to the server:

   1. On the client system, send a test message:

      # logger test

   2. On the server system, view the /var/log/<host2.example.com>/messages log, for example:

      # cat /var/log/<host2.example.com>/messages
      Aug  5 13:48:31 <host2.example.com> root[6778]: test

      Where <host2.example.com> is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.