Risk reduction and recovery operations

Procedure

1. Install the ReaR utility:

   # dnf install rear

   Copy to Clipboard Toggle word wrap

2. Modify the ReaR configuration file in an editor of your choice, for example:

   # vi /etc/rear/local.conf

   Copy to Clipboard Toggle word wrap

3. Add the backup setting details to /etc/rear/local.conf. For example, in the case of the NETFS backup method, add the following lines:

   BACKUP=NETFS
   BACKUP_URL=backup.location

   Copy to Clipboard Toggle word wrap

   Replace backup.location with the URL of your backup location.

4. To configure ReaR to keep the previous backup archive when the new one is created, also add the following line to the configuration file:

   NETFS_KEEP_OLD_BACKUP_COPY=y

   Copy to Clipboard Toggle word wrap

5. To make the backups incremental, meaning that only the changed files are backed up on each run, add the following line:

   BACKUP_TYPE=incremental

   Copy to Clipboard Toggle word wrap

6. Create a rescue system:

   # rear mkrescue

   Copy to Clipboard Toggle word wrap

7. Create a backup according to the restore plan. For example, in the case of the NETFS backup method, enter:

   # rear mkbackuponly

   Copy to Clipboard Toggle word wrap

   Alternatively, you can create the rescue system and the backup in a single step by running the following command:

   # rear mkbackup

   Copy to Clipboard Toggle word wrap

   This command combines the functionality of the rear mkrescue and rear mkbackuponly commands.

Procedure

1. Add the following variables to the /etc/rear/local.conf to configure ReaR for producing a rescue image on the 64-bit IBM Z architecture:

   1. To configure the IPL output method, add OUTPUT=IPL.

   2. To configure the backup method and destination, add BACKUP and BACKUP_URL variables. For example:

      BACKUP=NETFS
      
      BACKUP_URL=nfs://<nfsserver_name>/<share_path>

      Copy to Clipboard Toggle word wrap
      Important

      The local backup storage is currently not supported on the 64-bit IBM Z architecture.

   3. Optional: You can also configure the OUTPUT_URL variable to save the kernel and initrd files. By default, the OUTPUT_URL is aligned with BACKUP_URL.

2. To perform backup and rescue image creation:

   # rear mkbackup

   Copy to Clipboard Toggle word wrap

3. This creates the kernel and initrd files at the location specified by the BACKUP_URL or OUTPUT_URL (if set) variable, and a backup by using the specified backup method.

   Warning

   The rescue process reformats all the DASDs (Direct Attached Storage Devices) connected to the system. Do not attempt a system recovery if there is any valuable data present on the system storage devices. This also includes the device prepared with the zipl boot loader, ReaR kernel, and initrd that were used to boot into the rescue environment. Ensure to keep a copy.

4. To recover the system, use the previously created ReaR kernel and initrd files, and boot from a Direct Attached Storage Device (DASD) or a Fibre Channel Protocol (FCP)-attached SCSI device prepared with the zipl boot loader, kernel, and initrd.
5. When the rescue kernel and initrd get booted, it starts the ReaR rescue environment. Proceed with system recovery.

Procedure

1. Log in to the RHEL 10 web console.
2. Click Logs.
3. Open log entry details by clicking on your selected log entry in the list.

Procedure

1. Log in to the RHEL 10 web console.
2. Click Logs.
3. Click Toggle filters.
4. To change the default log filtering, use the Time, Priority, and Identifier drop-down menus.

5. Optional: By default, the web console shows the latest log entries. To filter by a specific time range, click the Expand button.

   

6. Click the → button (right pointing arrow) to apply your filters.
7. To open a log entry, click the selected log entry.

Procedure

1. Log in to the RHEL 10 web console.
2. Click Logs.

3. Use the drop-down menus to specify the three main quantifiers - time range, priority, and identifier(s) - you want to filter.

   The Priority quantifier must have a value. If you do not specify a value, it automatically filters the Error and above priority. The options you set reflect in the text search box.

4. Specify the log field you want to filter.

   You can add several log fields.

5. You can use a free-form string to search for anything else. The search box also accepts regular expressions.

Procedure

1. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Configure journald
     hosts: managed-node-01.example.com
     tasks:
       - name: Configure persistent logging
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.journald
         vars:
           journald_persistent: true
           journald_max_disk_size: <size>
           journald_per_user: true
           journald_sync_interval: <interval>

   Copy to Clipboard Toggle word wrap

   The settings specified in the example playbook include the following:

   journald_persistent: true

   Enables persistent logging.

   journald_max_disk_size: <size>

   Specifies the maximum size of disk space for journal files in MB, for example, 2048.

   journald_per_user: true

   Configures journald to keep log data separate for each user.

   journald_sync_interval: <interval>

   Sets the synchronization interval in minutes, for example, 1.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.journald/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Copy to Clipboard Toggle word wrap

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

   Copy to Clipboard Toggle word wrap

Procedure

1. Optional: To use a different port for rsyslog traffic, add the syslogd_port_t SELinux type to port. For example, enable port 30514:

   # semanage port -a -t syslogd_port_t -p tcp 30514

   Copy to Clipboard Toggle word wrap

2. Optional: To use a different port for rsyslog traffic, configure firewalld to allow incoming rsyslog traffic on that port. For example, allow TCP traffic on port 30514:

   # firewall-cmd --zone=<zone_name> --permanent --add-port=30514/tcp
   success
   # firewall-cmd --reload

   Copy to Clipboard Toggle word wrap

3. Create a new file in the /etc/rsyslog.d/ directory named, for example, remotelog.conf, and insert the following content:

   # Define templates before the rules that use them
   # Per-Host templates for remote systems
   template(name="TmplAuthpriv" type="list") {
       constant(value="/var/log/remote/auth/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   template(name="TmplMsg" type="list") {
       constant(value="/var/log/remote/msg/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   # Provides TCP syslog reception
   module(load="imtcp")
   
   # Adding this ruleset to process remote messages
   ruleset(name="remote1"){
        authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
         *.info;mail.none;authpriv.none;cron.none
   action(type="omfile" DynaFile="TmplMsg")
   }
   
   input(type="imtcp" port="30514" ruleset="remote1")

   Copy to Clipboard Toggle word wrap
4. Save the changes to the /etc/rsyslog.d/remotelog.conf file.

5. Test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-2.el8, config validation run...
   rsyslogd: End of config validation run. Bye.

   Copy to Clipboard Toggle word wrap

6. Make sure the rsyslog service is running and enabled on the logging server:

   # systemctl status rsyslog

   Copy to Clipboard Toggle word wrap

7. Restart the rsyslog service.

   # systemctl restart rsyslog

   Copy to Clipboard Toggle word wrap

8. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a new file in the /etc/rsyslog.d/ directory named, for example, 10-remotelog.conf, and insert the following content:

   *.* action(type="omfwd"
         queue.type="linkedlist"
         queue.filename="example_fwd"
         action.resumeRetryCount="-1"
         queue.saveOnShutdown="on"
         target="example.com" port="30514" protocol="tcp"
        )

   Copy to Clipboard Toggle word wrap

   Where:

   • The queue.type="linkedlist" setting enables a LinkedList in-memory queue,
   • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
   • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
   • The queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.

   • The last line forwards all received messages to the logging server. Port specification is optional.

     With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

     Note

     Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

2. Restart the rsyslog service.

   # systemctl restart rsyslog

   Copy to Clipboard Toggle word wrap

1. On the client system, send a test message:

   # logger test

   Copy to Clipboard Toggle word wrap

2. On the server system, view the /var/log/messages log, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Copy to Clipboard Toggle word wrap

   Where hostname is the hostname of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Configure the server for receiving encrypted logs from your client systems:

   1. Create a new file in the /etc/rsyslog.d/ directory named, for example, securelogser.conf.

   2. To encrypt the communication, the configuration file must contain paths to certificate files on your server, a selected authentication method, and a stream driver that supports TLS encryption. Add the following lines to the /etc/rsyslog.d/securelogser.conf file:

      # Set certificate files
      global(
        DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
        DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/server-cert.pem"
        DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/server-key.pem"
      )
      
      # TCP listener
      module(
        load="imtcp"
        PermittedPeer=["client1.example.com", "client2.example.com"]
        StreamDriver.AuthMode="x509/name"
        StreamDriver.Mode="1"
        StreamDriver.Name="ossl"
      )
      
      # Start up listener at port 514
      input(
        type="imtcp"
        port="514"
      )

      Copy to Clipboard Toggle word wrap
      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option. See the documentation installed with the rsyslog-doc package for more information about less strict authentication modes than x509/name.

   3. Optional: To customize the connection configuration, replace the input section with the following:

      input(
        type="imtcp"
        Port="50515"
        StreamDriver.Name="<driver>"
        streamdriver.CAFile="/etc/rsyslog.d/<ca1>.pem"
        streamdriver.CertFile="/etc/rsyslog.d/<server1_cert>.pem"
        streamdriver.KeyFile="/etc/rsyslog.d/<server1_key>.pem"
      )

      Copy to Clipboard Toggle word wrap
      • Replace <driver> with ossl or gtls depending on the driver you want to use.
      • Replace <ca1> with the CA certificate, <server1_cert> with the certificate, and <server1_key> with the key of the customized connection.
   4. Save the changes to the /etc/rsyslog.d/securelogser.conf file.

   5. Verify the syntax of the /etc/rsyslog.conf file and any files in the /etc/rsyslog.d/ directory:

      # rsyslogd -N 1
      rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
      rsyslogd: End of config validation run. Bye.

      Copy to Clipboard Toggle word wrap

   6. Make sure the rsyslog service is running and enabled on the logging server:

      # systemctl status rsyslog

      Copy to Clipboard Toggle word wrap

   7. Restart the rsyslog service:

      # systemctl restart rsyslog

      Copy to Clipboard Toggle word wrap

   8. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

      Copy to Clipboard Toggle word wrap

2. Configure clients for sending encrypted logs to the server:

   1. On a client system, create a new file in the /etc/rsyslog.d/ directory named, for example, securelogcli.conf.

   2. Add the following lines to the /etc/rsyslog.d/securelogcli.conf file:

      # Set certificate files
      global(
        DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
        DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/client-cert.pem"
        DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/client-key.pem"
      )
      
      
      # Set up the action for all messages
      *.* action(
        type="omfwd"
        StreamDriver="ossl"
        StreamDriverMode="1"
        StreamDriverPermittedPeers="server.example.com"
        StreamDriverAuthMode="x509/name"
        target="server.example.com" port="514" protocol="tcp"
      )

      Copy to Clipboard Toggle word wrap
      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option.

   3. Optional: To customize the connection configuration, replace the action section with the following:

      local1.* action(
        type="omfwd"
        StreamDriver="<driver>"
        StreamDriverMode="1"
        StreamDriverAuthMode="x509/certvalid"
        streamDriver.CAFile="/etc/rsyslog.d/<ca1>.pem"
        streamDriver.CertFile="/etc/rsyslog.d/<client1_cert>.pem"
        streamDriver.KeyFile="/etc/rsyslog.d/<client1_key>.pem"
        target="server.example.com" port="514" protocol="tcp"
        )

      Copy to Clipboard Toggle word wrap
      • Replace <driver> with ossl or gtls depending on the driver you want to use.
      • Replace <ca1> with the CA certificate, <client1_cert> with the certificate, and <client1_key> with the key of the customized connection.
   4. Save the changes to the /etc/rsyslog.d/securelogcli.conf file.

   5. Verify the syntax of the /etc/rsyslog.conf file and other files in the /etc/rsyslog.d/ directory:

      # rsyslogd -N 1
      rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
      rsyslogd: End of config validation run. Bye.

      Copy to Clipboard Toggle word wrap

   6. Make sure the rsyslog service is running and enabled on the logging server:

      # systemctl status rsyslog

      Copy to Clipboard Toggle word wrap

   7. Restart the rsyslog service:

      # systemctl restart rsyslog

      Copy to Clipboard Toggle word wrap

   8. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

      Copy to Clipboard Toggle word wrap

1. On the client system, send a test message:

   # logger test

   Copy to Clipboard Toggle word wrap

2. On the server system, view the /var/log/messages log, for example:

   # cat /var/log/remote/msg/<hostname>/root.log
   Feb 25 03:53:17 <hostname> root[6064]: test

   Copy to Clipboard Toggle word wrap

   Where <hostname> is the hostname of the client system. Note that the log contains the user name of the user who entered the logger command, in this case, root.

Procedure

1. Optional: To use a different port for rsyslog traffic than the default port 514:

   1. Add the syslogd_port_t SELinux type to the SELinux policy configuration, replacing portno with the port number you want rsyslog to use:

      # semanage port -a -t syslogd_port_t -p udp portno

      Copy to Clipboard Toggle word wrap

   2. Configure firewalld to allow incoming rsyslog traffic, replacing portno with the port number and zone with the zone you want rsyslog to use:

      # firewall-cmd --zone=zone --permanent --add-port=portno/udp
      success
      # firewall-cmd --reload

      Copy to Clipboard Toggle word wrap

   3. Reload the firewall rules:

      # firewall-cmd --reload

      Copy to Clipboard Toggle word wrap

2. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, remotelogserv.conf, and insert the following content:

   # Define templates before the rules that use them
   # Per-Host templates for remote systems
   template(name="TmplAuthpriv" type="list") {
       constant(value="/var/log/remote/auth/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   template(name="TmplMsg" type="list") {
       constant(value="/var/log/remote/msg/")
       property(name="hostname")
       constant(value="/")
       property(name="programname" SecurePath="replace")
       constant(value=".log")
       }
   
   # Provides UDP syslog reception
   module(load="imudp")
   
   # This ruleset processes remote messages
   ruleset(name="remote1"){
        authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
         *.info;mail.none;authpriv.none;cron.none
   action(type="omfile" DynaFile="TmplMsg")
   }
   
   input(type="imudp" port="514" ruleset="remote1")

   Copy to Clipboard Toggle word wrap

   Where 514 is the port number rsyslog uses by default. You can specify a different port instead.

3. Verify the syntax of the /etc/rsyslog.conf file and all .conf files in the /etc/rsyslog.d/ directory:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-2.el8, config validation run...

   Copy to Clipboard Toggle word wrap

4. Restart the rsyslog service.

   # systemctl restart rsyslog

   Copy to Clipboard Toggle word wrap

5. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, 10-remotelogcli.conf, and insert the following content:

   *.* action(type="omfwd"
         queue.type="linkedlist"
         queue.filename="example_fwd"
         action.resumeRetryCount="-1"
         queue.saveOnShutdown="on"
         target="example.com" port="portno" protocol="udp"
        )

   Copy to Clipboard Toggle word wrap

   Where:

   • The queue.type="linkedlist" setting enables a LinkedList in-memory queue.
   • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
   • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if the server is not responding.
   • The enabled queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.
   • The portno value is the port number you want rsyslog to use. The default value is 514.

   • The last line forwards all received messages to the logging server, port specification is optional.

     With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

   Note

   Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

2. Restart the rsyslog service.

   # systemctl restart rsyslog

   Copy to Clipboard Toggle word wrap

3. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

   # systemctl enable rsyslog

   Copy to Clipboard Toggle word wrap

1. On the client system, send a test message:

   # logger test

   Copy to Clipboard Toggle word wrap

2. On the server system, view the /var/log/remote/msg/hostname/root.log log, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Copy to Clipboard Toggle word wrap

   Where hostname is the hostname of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Configure the client system for reliable remote logging:

   1. On the client system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpclient.conf, and insert the following content:

      module(load="omrelp")
      *.* action(type="omrelp" target="_target_IP_" port="_target_port_")

      Copy to Clipboard Toggle word wrap

      Where:

      • target_IP is the IP address of the logging server.
      • target_port is the port of the logging server.
   2. Save the changes to the /etc/rsyslog.d/relpclient.conf file.

   3. Restart the rsyslog service.

      # systemctl restart rsyslog

      Copy to Clipboard Toggle word wrap

   4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

      Copy to Clipboard Toggle word wrap

2. Configure the server system for reliable remote logging:

   1. On the server system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpserv.conf, and insert the following content:

      ruleset(name="relp"){
      *.* action(type="omfile" file="_log_path_")
      }
      
      
      module(load="imrelp")
      input(type="imrelp" port="_target_port_" ruleset="relp")

      Copy to Clipboard Toggle word wrap

      Where:

      • log_path specifies the path for storing messages.
      • target_port is the port of the logging server. Use the same value as in the client configuration file.
   2. Save the changes to the /etc/rsyslog.d/relpserv.conf file.

   3. Restart the rsyslog service.

      # systemctl restart rsyslog

      Copy to Clipboard Toggle word wrap

   4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

      Copy to Clipboard Toggle word wrap

1. On the client system, send a test message:

   # logger test

   Copy to Clipboard Toggle word wrap

2. On the server system, view the log at the specified log_path, for example:

   # cat /var/log/remote/msg/hostname/root.log
   Feb 25 03:53:17 hostname root[6064]: test

   Copy to Clipboard Toggle word wrap

   Where hostname is the hostname of the client system. Note that the log contains the user name of the user who entered the logger command, in this case, root.

Procedure

1. Install the netconsole-service package:

   # dnf install netconsole-service

   Copy to Clipboard Toggle word wrap

2. Edit the /etc/sysconfig/netconsole file and set the SYSLOGADDR parameter to the IP address of the remote host:

   # SYSLOGADDR=192.0.2.1

   Copy to Clipboard Toggle word wrap

3. Enable and start the netconsole service:

   # systemctl enable --now netconsole

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Deploy the logging solution
     hosts: managed-node-01.example.com
     tasks:
       - name: Filter logs based on a specific value they contain
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.logging
         vars:
           logging_inputs:
             - name: files_input
               type: basics
           logging_outputs:
             - name: files_output0
               type: files
               property: msg
               property_op: contains
               property_value: error
               path: /var/log/errors.log
             - name: files_output1
               type: files
               property: msg
               property_op: "!contains"
               property_value: error
               path: /var/log/others.log
           logging_flows:
             - name: flow0
               inputs: [files_input]
               outputs: [files_output0, files_output1]

   Copy to Clipboard Toggle word wrap

   The settings specified in the example playbook include the following:

   logging_inputs

   Defines a list of logging input dictionaries. The type: basics option covers inputs from systemd journal or Unix socket.

   logging_outputs

   Defines a list of logging output dictionaries. The type: files option supports storing logs in the local files, usually in the /var/log/ directory. The property: msg; property: contains; and property_value: error options specify that all logs that contain the error string are stored in the /var/log/errors.log file. The property: msg; property: !contains; and property_value: error options specify that all other logs are put in the /var/log/others.log file. You can replace the error value with the string by which you want to filter.

   logging_flows

   Defines a list of logging flow dictionaries to specify relationships between logging_inputs and logging_outputs. The inputs: [files_input] option specifies a list of inputs, from which processing of logs starts. The outputs: [files_output0, files_output1] option specifies a list of outputs, to which the logs are sent.

   For details about all variables used in the playbook and more information about rsyslog, see the /usr/share/ansible/roles/rhel-system-roles.logging/README.md file and rsyslog.conf(5) and syslog(3) manual pages on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Copy to Clipboard Toggle word wrap

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

   Copy to Clipboard Toggle word wrap

Verification

1. On the managed node, test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-6.el8, config validation run...
   rsyslogd: End of config validation run. Bye.

   Copy to Clipboard Toggle word wrap

2. On the managed node, verify that the system sends messages that contain the error string to the log:

   1. Send a test message:

      # logger error

      Copy to Clipboard Toggle word wrap

   2. View the /var/log/errors.log log, for example:

      # cat /var/log/errors.log
      Aug  5 13:48:31 hostname root[6778]: error

      Copy to Clipboard Toggle word wrap

      Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Deploy the logging solution
     hosts: managed-node-01.example.com
     tasks:
       - name: Configure the server to receive remote input
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.logging
         vars:
           logging_inputs:
             - name: remote_udp_input
               type: remote
               udp_ports: [ 601 ]
             - name: remote_tcp_input
               type: remote
               tcp_ports: [ 601 ]
           logging_outputs:
             - name: remote_files_output
               type: remote_files
           logging_flows:
             - name: flow_0
               inputs: [remote_udp_input, remote_tcp_input]
               outputs: [remote_files_output]
   
   - name: Deploy the logging solution
     hosts: managed-node-02.example.com
     tasks:
       - name: Configure the server to output the logs to local files in directories named by remote host names
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.logging
         vars:
           logging_inputs:
             - name: basic_input
               type: basics
           logging_outputs:
             - name: forward_output0
               type: forwards
               severity: info
               target: <host1.example.com>
               udp_port: 601
             - name: forward_output1
               type: forwards
               facility: mail
               target: <host1.example.com>
               tcp_port: 601
           logging_flows:
             - name: flows0
               inputs: [basic_input]
               outputs: [forward_output0, forward_output1]

   Copy to Clipboard Toggle word wrap

   The settings specified in the first play of the example playbook include the following:

   logging_inputs

   Defines a list of logging input dictionaries. The type: remote option covers remote inputs from the other logging system over the network. The udp_ports: [ 601 ] option defines a list of UDP port numbers to monitor. The tcp_ports: [ 601 ] option defines a list of TCP port numbers to monitor. If both udp_ports and tcp_ports are set, udp_ports is used and tcp_ports is dropped.

   logging_outputs

   Defines a list of logging output dictionaries. The type: remote_files option makes output store logs to the local files per remote host and program name originated the logs.

   logging_flows

   Defines a list of logging flow dictionaries to specify relationships between logging_inputs and logging_outputs. The inputs: [remote_udp_input, remote_tcp_input] option specifies a list of inputs, from which processing of logs starts. The outputs: [remote_files_output] option specifies a list of outputs, to which the logs are sent.

   The settings specified in the second play of the example playbook include the following:

   logging_inputs

   Defines a list of logging input dictionaries. The type: basics option covers inputs from systemd journal or Unix socket.

   logging_outputs

   Defines a list of logging output dictionaries. The type: forwards option supports sending logs to the remote logging server over the network. The severity: info option refers to log messages of informative importance. The facility: mail option refers to the type of system program that is generating the log message. The target: <host1.example.com> option specifies the hostname of the remote logging server. The udp_port: 601/tcp_port: 601 options define the UDP/TCP ports on which the remote logging server listens.

   logging_flows

   Defines a list of logging flow dictionaries to specify relationships between logging_inputs and logging_outputs. The inputs: [basic_input] option specifies a list of inputs, from which processing of logs starts. The outputs: [forward_output0, forward_output1] option specifies a list of outputs, to which the logs are sent.

   For details about the role variables and more information about rsyslog, see the /usr/share/ansible/roles/rhel-system-roles.logging/README.md file and the rsyslog.conf(5) and syslog(3) manual pages on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Copy to Clipboard Toggle word wrap

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

   Copy to Clipboard Toggle word wrap

Verification

1. On both the client and the server system, test the syntax of the /etc/rsyslog.conf file:

   # rsyslogd -N 1
   rsyslogd: version 8.1911.0-6.el8, config validation run (level 1), master config /etc/rsyslog.conf
   rsyslogd: End of config validation run. Bye.

   Copy to Clipboard Toggle word wrap

2. Verify that the client system sends messages to the server:

   1. On the client system, send a test message:

      # logger test

      Copy to Clipboard Toggle word wrap

   2. On the server system, view the /var/log/<host2.example.com>/messages log, for example:

      # cat /var/log/<host2.example.com>/messages
      Aug  5 13:48:31 <host2.example.com> root[6778]: test

      Copy to Clipboard Toggle word wrap

      Where <host2.example.com> is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Procedure

1. Copy the /usr/lib/systemd/system/auditd.service file to the /etc/systemd/system/ directory:

   # cp -f /usr/lib/systemd/system/auditd.service /etc/systemd/system/

   Copy to Clipboard Toggle word wrap

2. Edit the /etc/systemd/system/auditd.service file in a text editor of your choice, for example:

   # vi /etc/systemd/system/auditd.service

   Copy to Clipboard Toggle word wrap

3. Comment out the line containing augenrules, and uncomment the line containing the auditctl -R command:

   #ExecStartPost=-/sbin/augenrules --load
   ExecStartPost=-/sbin/auditctl -R /etc/audit/audit.rules

   Copy to Clipboard Toggle word wrap

4. Reload the systemd daemon to fetch changes in the auditd.service file:

   # systemctl daemon-reload

   Copy to Clipboard Toggle word wrap

5. Restart the auditd service:

   # service auditd restart

   Copy to Clipboard Toggle word wrap

Procedure

1. Copy the pre-configured rule file 44-installers.rules from the /usr/share/audit/sample-rules/ directory to the /etc/audit/rules.d/ directory:

   # cp /usr/share/audit/sample-rules/44-installers.rules /etc/audit/rules.d/

   Copy to Clipboard Toggle word wrap

2. Load the audit rules:

   # augenrules --load

   Copy to Clipboard Toggle word wrap

Verification

1. List the loaded rules:

   # auditctl -l
   -p x-w /usr/bin/dnf-3 -k software-installer
   -p x-w /usr/bin/yum -k software-installer
   -p x-w /usr/bin/pip -k software-installer
   -p x-w /usr/bin/npm -k software-installer
   -p x-w /usr/bin/cpan -k software-installer
   -p x-w /usr/bin/gem -k software-installer
   -p x-w /usr/bin/luarocks -k software-installer

   Copy to Clipboard Toggle word wrap

2. Perform an installation, for example:

   # dnf reinstall -y vim-enhanced

   Copy to Clipboard Toggle word wrap

3. Search the Audit log for recent installation events, for example:

   # ausearch -ts recent -k software-installer
   ––––
   time->Thu Dec 16 10:33:46 2021
   type=PROCTITLE msg=audit(1639668826.074:298): proctitle=2F7573722F6C6962657865632F706C6174666F726D2D707974686F6E002F7573722F62696E2F646E66007265696E7374616C6C002D790076696D2D656E68616E636564
   type=PATH msg=audit(1639668826.074:298): item=2 name="/lib64/ld-linux-x86-64.so.2" inode=10092 dev=fd:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
   type=PATH msg=audit(1639668826.074:298): item=1 name="/usr/libexec/platform-python" inode=4618433 dev=fd:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:bin_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
   type=PATH msg=audit(1639668826.074:298): item=0 name="/usr/bin/dnf" inode=6886099 dev=fd:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:rpm_exec_t:s0 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
   type=CWD msg=audit(1639668826.074:298): cwd="/root"
   type=EXECVE msg=audit(1639668826.074:298): argc=5 a0="/usr/libexec/platform-python" a1="/usr/bin/dnf" a2="reinstall" a3="-y" a4="vim-enhanced"
   type=SYSCALL msg=audit(1639668826.074:298): arch=c000003e syscall=59 success=yes exit=0 a0=55c437f22b20 a1=55c437f2c9d0 a2=55c437f2aeb0 a3=8 items=3 ppid=5256 pid=5375 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=3 comm="dnf" exe="/usr/libexec/platform-python3.6" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="software-installer"

   Copy to Clipboard Toggle word wrap

Procedure

1. Search the Audit log for the USER_LOGIN message type:

   # ausearch -m USER_LOGIN -ts '12/02/2020' '18:00:00' -sv no
   time->Mon Nov 22 07:33:22 2021
   type=USER_LOGIN msg=audit(1637584402.416:92): pid=1939 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login acct="(unknown)" exe="/usr/sbin/sshd" hostname=? addr=10.37.128.108 terminal=ssh res=failed'

   Copy to Clipboard Toggle word wrap
   • You can specify the date and time with the -ts option. If you do not use this option, ausearch provides results from today, and if you omit time, ausearch provides results from midnight.
   • You can use the -sv yes option to filter out successful login attempts and -sv no for unsuccessful login attempts.

2. Pipe the raw output of the ausearch command into the aulast utility, which displays the output in a format similar to the output of the last command. For example:

   # ausearch --raw | aulast --stdin
   root     ssh          10.37.128.108    Mon Nov 22 07:33 - 07:33  (00:00)
   root     ssh          10.37.128.108    Mon Nov 22 07:33 - 07:33  (00:00)
   root     ssh          10.22.16.106     Mon Nov 22 07:40 - 07:40  (00:00)
   reboot   system boot  4.18.0-348.6.el8 Mon Nov 22 07:33

   Copy to Clipboard Toggle word wrap

3. Display the list of login events by using the aureport command with the --login -i options.

   # aureport --login -i
   
   Login Report
   ============================================
   # date time auid host term exe success event
   ============================================
   1. 11/16/2021 13:11:30 root 10.40.192.190 ssh /usr/sbin/sshd yes 6920
   2. 11/16/2021 13:11:31 root 10.40.192.190 ssh /usr/sbin/sshd yes 6925
   3. 11/16/2021 13:11:31 root 10.40.192.190 ssh /usr/sbin/sshd yes 6930
   4. 11/16/2021 13:11:31 root 10.40.192.190 ssh /usr/sbin/sshd yes 6935
   5. 11/16/2021 13:11:33 root 10.40.192.190 ssh /usr/sbin/sshd yes 6940
   6. 11/16/2021 13:11:33 root 10.40.192.190 /dev/pts/0 /usr/sbin/sshd yes 6945

   Copy to Clipboard Toggle word wrap