SupportConsoleDevelopersStart a trialContact

Chapter 12. Configuring logging by using RHEL System Roles

As a system administrator, you can use the logging System Role to configure a RHEL host as a logging server to collect logs from many client systems.

12.1. The logging System Role

With the logging System Role, you can deploy logging configurations on local and remote hosts.

To apply a logging System Role on one or more systems, you define the logging configuration in a playbook. A playbook is a list of one or more plays. Playbooks are human-readable, and they are written in the YAML format. For more information about playbooks, see Working with playbooks in Ansible documentation.

The set of systems that you want to configure according to the playbook is defined in an inventory file. For more information on creating and using inventories, see How to build your inventory in Ansible documentation.

Logging solutions provide multiple ways of reading logs and multiple logging outputs.

For example, a logging system can receive the following inputs:

  • local files,
  • systemd/journal,
  • another logging system over the network.

In addition, a logging system can have the following outputs:

  • logs stored in the local files in the /var/log directory,
  • logs sent to Elasticsearch,
  • logs forwarded to another logging system.

With the logging System Role, you can combine the inputs and outputs to fit your scenario. For example, you can configure a logging solution that stores inputs from journal in a local file, whereas inputs read from files are both forwarded to another logging system and stored in the local log files.

12.2. logging System Role parameters

In a logging System Role playbook, you define the inputs in the logging_inputs parameter, outputs in the logging_outputs parameter, and the relationships between the inputs and outputs in the logging_flows parameter. The logging System Role processes these variables with additional options to configure the logging system. You can also enable encryption or an automatic port management.

Note

Currently, the only available logging system in the logging System Role is Rsyslog.

  • logging_inputs: List of inputs for the logging solution.

    • name: Unique name of the input. Used in the logging_flows: inputs list and a part of the generated config file name.

    • type: Type of the input element. The type specifies a task type which corresponds to a directory name in roles/rsyslog/{tasks,vars}/inputs/.

      • basics: Inputs configuring inputs from systemd journal or unix socket.

        • kernel_message: Load imklog if set to true. Default to false.
        • use_imuxsock: Use imuxsock instead of imjournal. Default to false.
        • ratelimit_burst: Maximum number of messages that can be emitted within ratelimit_interval. Default to 20000 if use_imuxsock is false. Default to 200 if use_imuxsock is true.
        • ratelimit_interval: Interval to evaluate ratelimit_burst. Default to 600 seconds if use_imuxsock is false. Default to 0 if use_imuxsock is true. 0 indicates rate limiting is turned off.
        • persist_state_interval: Journal state is persisted every value messages. Default to 10. Effective only when use_imuxsock is false.
      • files: Inputs configuring inputs from local files.
      • remote: Inputs configuring inputs from the other logging system over network.
    • state: State of the configuration file. present or absent. Default to present.

  • logging_outputs: List of outputs for the logging solution.

    • files: Outputs configuring outputs to local files.
    • forwards: Outputs configuring outputs to another logging system.
    • remote_files: Outputs configuring outputs from another logging system to local files.

  • logging_flows: List of flows that define relationships between logging_inputs and logging_outputs. The logging_flows variable has the following keys:

    • name: Unique name of the flow
    • inputs: List of logging_inputs name values
    • outputs: List of logging_outputs name values.
  • logging_manage_firewall: If set to true, the variable uses the firewall role to automatically manage port access from within the logging role.
  • logging_manage_selinux: If set to true, the variable uses the selinux role to automatically manage port access from within the logging role.

Additional resources

  • Documentation installed with the rhel-system-roles package in /usr/share/ansible/roles/rhel-system-roles.logging/README.html

12.3. Applying a local logging System Role

Prepare and apply an Ansible playbook to configure a logging solution on a set of separate machines. Each machine records logs locally.

Prerequisites

  • Access and permissions to one or more managed nodes, which are systems you want to configure with the logging System Role.

  • Access and permissions to a control node, which is a system from which Red Hat Ansible Core configures other systems.

    On the control node:

    • The ansible-core and rhel-system-roles packages are installed.
Important

RHEL 8.0-8.5 provided access to a separate Ansible repository that contains Ansible Engine 2.9 for automation based on Ansible. Ansible Engine contains command-line utilities such as ansible, ansible-playbook, connectors such as docker and podman, and many plugins and modules. For information about how to obtain and install Ansible Engine, see the How to download and install Red Hat Ansible Engine Knowledgebase article.

RHEL 8.6 and 9.0 have introduced Ansible Core (provided as the ansible-core package), which contains the Ansible command-line utilities, commands, and a small set of built-in Ansible plugins. RHEL provides this package through the AppStream repository, and it has a limited scope of support. For more information, see the Scope of support for the Ansible Core package included in the RHEL 9 and RHEL 8.6 and later AppStream repositories Knowledgebase article.

  • An inventory file which lists the managed nodes.
Note

You do not have to have the rsyslog package installed, because the System Role installs rsyslog when deployed.

Procedure

  1. Create a playbook that defines the required role:

    1. Create a new YAML file and open it in a text editor, for example: 

      # vi logging-playbook.yml

    2. Insert the following content: 

      ---
- name: Deploying basics input and implicit files output
  hosts: all
  roles:
    - rhel-system-roles.logging
  vars:
    logging_inputs:
      - name: system_input
        type: basics
    logging_outputs:
      - name: files_output
        type: files
    logging_flows:
      - name: flow1
        inputs: [system_input]
        outputs: [files_output]

  2. Run the playbook on a specific inventory: 

    # ansible-playbook -i inventory-file /path/to/file/logging-playbook.yml

    Where:

    • inventory-file is the inventory file.
    • logging-playbook.yml is the playbook you use.

Verification

  1. Test the syntax of the /etc/rsyslog.conf file: 

    # rsyslogd -N 1
rsyslogd: version 8.1911.0-6.el8, config validation run...
rsyslogd: End of config validation run. Bye.

  2. Verify that the system sends messages to the log:

    1. Send a test message: 

      # logger test

    2. View the /var/log/messages log, for example: 

      # cat /var/log/messages
Aug  5 13:48:31 hostname root[6778]: test

      Where `hostname` is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

12.4. Filtering logs in a local logging System Role

You can deploy a logging solution which filters the logs based on the rsyslog property-based filter.

Prerequisites

  • Access and permissions to one or more managed nodes, which are systems you want to configure with the logging System Role.

  • Access and permissions to a control node, which is a system from which Red Hat Ansible Core configures other systems.

    On the control node:

    • Red Hat Ansible Core is installed
    • The rhel-system-roles package is installed
    • An inventory file which lists the managed nodes.
Note

You do not have to have the rsyslog package installed, because the System Role installs rsyslog when deployed.

Procedure

  1. Create a new playbook.yml file with the following content: 

    ---
- name: Deploying files input and configured files output
  hosts: all
  roles:
    - linux-system-roles.logging
  vars:
    logging_inputs:
      - name: files_input
        type: basics
    logging_outputs:
      - name: files_output0
        type: files
        property: msg
        property_op: contains
        property_value: error
        path: /var/log/errors.log
      - name: files_output1
        type: files
        property: msg
        property_op: "!contains"
        property_value: error
        path: /var/log/others.log
    logging_flows:
      - name: flow0
        inputs: [files_input]
        outputs: [files_output0, files_output1]

    Using this configuration, all messages that contain the error string are logged in /var/log/errors.log, and all other messages are logged in /var/log/others.log.

    You can replace the error property value with the string by which you want to filter.

    You can modify the variables according to your preferences.

  2. Optional: Verify playbook syntax. 

    # ansible-playbook --syntax-check playbook.yml

  3. Run the playbook on your inventory file: 

    # ansible-playbook -i inventory_file /path/to/file/playbook.yml

Verification

  1. Test the syntax of the /etc/rsyslog.conf file: 

    # rsyslogd -N 1
rsyslogd: version 8.1911.0-6.el8, config validation run...
rsyslogd: End of config validation run. Bye.

  2. Verify that the system sends messages that contain the error string to the log:

    1. Send a test message: 

      # logger error

    2. View the /var/log/errors.log log, for example: 

      # cat /var/log/errors.log
Aug  5 13:48:31 hostname root[6778]: error

      Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • Documentation installed with the rhel-system-roles package in /usr/share/ansible/roles/rhel-system-roles.logging/README.html

12.5. Applying a remote logging solution using the logging System Role

Follow these steps to prepare and apply a Red Hat Ansible Core playbook to configure a remote logging solution. In this playbook, one or more clients take logs from systemd-journal and forward them to a remote server. The server receives remote input from remote_rsyslog and remote_files and outputs the logs to local files in directories named by remote host names.

Prerequisites

  • Access and permissions to one or more managed nodes, which are systems you want to configure with the logging System Role.

  • Access and permissions to a control node, which is a system from which Red Hat Ansible Core configures other systems.

    On the control node:

    • The ansible-core and rhel-system-roles packages are installed.
    • An inventory file which lists the managed nodes.
Note

You do not have to have the rsyslog package installed, because the System Role installs rsyslog when deployed.

Procedure

  1. Create a playbook that defines the required role:

    1. Create a new YAML file and open it in a text editor, for example: 

      # vi logging-playbook.yml

    2. Insert the following content into the file: 

      ---
- name: Deploying remote input and remote_files output
  hosts: server
  roles:
    - rhel-system-roles.logging
  vars:
    logging_inputs:
      - name: remote_udp_input
        type: remote
        udp_ports: [ 601 ]
      - name: remote_tcp_input
        type: remote
        tcp_ports: [ 601 ]
    logging_outputs:
      - name: remote_files_output
        type: remote_files
    logging_flows:
      - name: flow_0
        inputs: [remote_udp_input, remote_tcp_input]
        outputs: [remote_files_output]

- name: Deploying basics input and forwards output
  hosts: clients
  roles:
    - rhel-system-roles.logging
  vars:
    logging_inputs:
      - name: basic_input
        type: basics
    logging_outputs:
      - name: forward_output0
        type: forwards
        severity: info
        target: _host1.example.com_
        udp_port: 601
      - name: forward_output1
        type: forwards
        facility: mail
        target: _host1.example.com_
        tcp_port: 601
    logging_flows:
      - name: flows0
        inputs: [basic_input]
        outputs: [forward_output0, forward_output1]

[basic_input]
[forward_output0, forward_output1]

      Where host1.example.com is the logging server.

      Note

      You can modify the parameters in the playbook to fit your needs.

      Warning

      The logging solution works only with the ports defined in the SELinux policy of the server or client system and open in the firewall. The default SELinux policy includes ports 601, 514, 6514, 10514, and 20514. To use a different port, modify the SELinux policy on the client and server systems.

  2. Create an inventory file that lists your servers and clients:

    1. Create a new file and open it in a text editor, for example: 

      # vi inventory.ini

    2. Insert the following content into the inventory file: 

      [servers]
server ansible_host=host1.example.com
[clients]
client ansible_host=host2.example.com

      Where:

      • host1.example.com is the logging server.
      • host2.example.com is the logging client.

  3. Run the playbook on your inventory. 

    # ansible-playbook -i /path/to/file/inventory.ini /path/to/file/_logging-playbook.yml

    Where:

    • inventory.ini is the inventory file.
    • logging-playbook.yml is the playbook you created.

Verification

  1. On both the client and the server system, test the syntax of the /etc/rsyslog.conf file: 

    # rsyslogd -N 1
rsyslogd: version 8.1911.0-6.el8, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

  2. Verify that the client system sends messages to the server:

    1. On the client system, send a test message: 

      # logger test

    2. On the server system, view the /var/log/messages log, for example: 

      # cat /var/log/messages
Aug  5 13:48:31 host2.example.com root[6778]: test

      Where host2.example.com is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

12.6. Using the logging System Role with TLS

Transport Layer Security (TLS) is a cryptographic protocol designed to securely communicate over the computer network.

As an administrator, you can use the logging RHEL System Role to configure secure transfer of logs using Red Hat Ansible Automation Platform.

12.6.1. Configuring client logging with TLS

You can use the logging System Role to configure logging in RHEL systems that are logged on a local machine and can transfer logs to the remote logging system with TLS by running an Ansible playbook.

This procedure configures TLS on all hosts in the clients group in the Ansible inventory. The TLS protocol encrypts the message transmission for secure transfer of logs over the network.

Prerequisites

  • You have permissions to run playbooks on managed nodes on which you want to configure TLS.
  • The managed nodes are listed in the inventory file on the control node.
  • The ansible and rhel-system-roles packages are installed on the control node.

Procedure

  1. Create a playbook.yml file with the following content: 

    ---
- name: Deploying files input and forwards output with certs
  hosts: clients
  roles:
    - rhel-system-roles.logging
  vars:
    logging_pki_files:
      - ca_cert_src: /local/path/to/ca_cert.pem
        cert_src: /local/path/to/cert.pem
        private_key_src: /local/path/to/key.pem
    logging_inputs:
      - name: input_name
        type: files
        input_log_path: /var/log/containers/*.log
    logging_outputs:
      - name: output_name
        type: forwards
        target: your_target_host
        tcp_port: 514
        tls: true
        pki_authmode: x509/name
        permitted_server: 'server.example.com'
    logging_flows:
      - name: flow_name
        inputs: [input_name]
        outputs: [output_name]

    The playbook uses the following parameters:

    logging_pki_files
    Using this parameter you can configure TLS and has to pass ca_cert_src, cert_src, and private_key_src parameters.
    ca_cert
    Represents the path to CA certificate. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    cert
    Represents the path to cert. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    private_key
    Represents the path to private key. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    ca_cert_src
    Represents local CA cert file path which is copied to the target host. If ca_cert is specified, it is copied to the location.
    cert_src
    Represents the local cert file path which is copied to the target host. If cert is specified, it is copied to the location.
    private_key_src
    Represents the local key file path which is copied to the target host. If private_key is specified, it is copied to the location.
    tls
    Using this parameter ensures secure transfer of logs over the network. If you do not want a secure wrapper, you can set tls: true.

  2. Verify playbook syntax: 

    # ansible-playbook --syntax-check playbook.yml

  3. Run the playbook on your inventory file: 

    # ansible-playbook -i inventory_file playbook.yml

12.6.2. Configuring server logging with TLS

You can use the logging System Role to configure logging in RHEL systems as a server and can receive logs from the remote logging system with TLS by running an Ansible playbook.

This procedure configures TLS on all hosts in the server group in the Ansible inventory.

Prerequisites

  • You have permissions to run playbooks on managed nodes on which you want to configure TLS.
  • The managed nodes are listed in the inventory file on the control node.
  • The ansible and rhel-system-roles packages are installed on the control node.

Procedure

  1. Create a playbook.yml file with the following content: 

    ---
- name: Deploying remote input and remote_files output with certs
  hosts: server
  roles:
    - rhel-system-roles.logging
  vars:
    logging_pki_files:
      - ca_cert_src: /local/path/to/ca_cert.pem
        cert_src: /local/path/to/cert.pem
        private_key_src: /local/path/to/key.pem
    logging_inputs:
      - name: input_name
        type: remote
        tcp_ports: 514
        tls: true
        permitted_clients: ['clients.example.com']
    logging_outputs:
      - name: output_name
        type: remote_files
        remote_log_path: /var/log/remote/%FROMHOST%/%PROGRAMNAME:::secpath-replace%.log
        async_writing: true
        client_count: 20
        io_buffer_size: 8192
    logging_flows:
      - name: flow_name
        inputs: [input_name]
        outputs: [output_name]

    The playbook uses the following parameters:

    logging_pki_files
    Using this parameter you can configure TLS and has to pass ca_cert_src, cert_src, and private_key_src parameters.
    ca_cert
    Represents the path to CA certificate. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    cert
    Represents the path to cert. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    private_key
    Represents the path to private key. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    ca_cert_src
    Represents local CA cert file path which is copied to the target host. If ca_cert is specified, it is copied to the location.
    cert_src
    Represents the local cert file path which is copied to the target host. If cert is specified, it is copied to the location.
    private_key_src
    Represents the local key file path which is copied to the target host. If private_key is specified, it is copied to the location.
    tls
    Using this parameter ensures secure transfer of logs over the network. If you do not want a secure wrapper, you can set tls: true.

  2. Verify playbook syntax: 

    # ansible-playbook --syntax-check playbook.yml

  3. Run the playbook on your inventory file: 

    # ansible-playbook -i inventory_file playbook.yml

12.7. Using the logging System Roles with RELP

Reliable Event Logging Protocol (RELP) is a networking protocol for data and message logging over the TCP network. It ensures reliable delivery of event messages and you can use it in environments that do not tolerate any message loss.

The RELP sender transfers log entries in form of commands and the receiver acknowledges them once they are processed. To ensure consistency, RELP stores the transaction number to each transferred command for any kind of message recovery.

You can consider a remote logging system in between the RELP Client and RELP Server. The RELP Client transfers the logs to the remote logging system and the RELP Server receives all the logs sent by the remote logging system.

Administrators can use the logging System Role to configure the logging system to reliably send and receive log entries.

12.7.1. Configuring client logging with RELP

You can use the logging System Role to configure logging in RHEL systems that are logged on a local machine and can transfer logs to the remote logging system with RELP by running an Ansible playbook.

This procedure configures RELP on all hosts in the clients group in the Ansible inventory. The RELP configuration uses Transport Layer Security (TLS) to encrypt the message transmission for secure transfer of logs over the network.

Prerequisites

  • You have permissions to run playbooks on managed nodes on which you want to configure RELP.
  • The managed nodes are listed in the inventory file on the control node.
  • The ansible and rhel-system-roles packages are installed on the control node.

Procedure

  1. Create a playbook.yml file with the following content: 

    ---
- name: Deploying basic input and relp output
  hosts: clients
  roles:
    - rhel-system-roles.logging
  vars:
    logging_inputs:
      - name: basic_input
        type: basics
    logging_outputs:
      - name: relp_client
        type: relp
        target: _logging.server.com_
        port: 20514
        tls: true
        ca_cert: _/etc/pki/tls/certs/ca.pem_
        cert: _/etc/pki/tls/certs/client-cert.pem_
        private_key: _/etc/pki/tls/private/client-key.pem_
        pki_authmode: name
        permitted_servers:
          - '*.server.example.com'
    logging_flows:
      - name: _example_flow_
        inputs: [basic_input]
        outputs: [relp_client]

    The playbooks uses following settings:

    • target: This is a required parameter that specifies the host name where the remote logging system is running.
    • port: Port number the remote logging system is listening.

    • tls: Ensures secure transfer of logs over the network. If you do not want a secure wrapper you can set the tls variable to false. By default tls parameter is set to true while working with RELP and requires key/certificates and triplets {ca_cert, cert, private_key} and/or {ca_cert_src, cert_src, private_key_src}.

      • If {ca_cert_src, cert_src, private_key_src} triplet is set, the default locations /etc/pki/tls/certs and /etc/pki/tls/private are used as the destination on the managed node to transfer files from control node. In this case, the file names are identical to the original ones in the triplet
      • If {ca_cert, cert, private_key} triplet is set, files are expected to be on the default path before the logging configuration.
      • If both the triplets are set, files are transferred from local path from control node to specific path of the managed node.
    • ca_cert: Represents the path to CA certificate. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    • cert: Represents the path to cert. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    • private_key: Represents the path to private key. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    • ca_cert_src: Represents local CA cert file path which is copied to the target host. If ca_cert is specified, it is copied to the location.
    • cert_src: Represents the local cert file path which is copied to the target host. If cert is specified, it is copied to the location.
    • private_key_src: Represents the local key file path which is copied to the target host. If private_key is specified, it is copied to the location.
    • pki_authmode: Accepts the authentication mode as name or fingerprint.
    • permitted_servers: List of servers that will be allowed by the logging client to connect and send logs over TLS.
    • inputs: List of logging input dictionary.
    • outputs: List of logging output dictionary.

  2. Optional: Verify playbook syntax. 

    # ansible-playbook --syntax-check playbook.yml

  3. Run the playbook: 

    # ansible-playbook -i inventory_file playbook.yml

12.7.2. Configuring server logging with RELP

You can use the logging System Role to configure logging in RHEL systems as a server and can receive logs from the remote logging system with RELP by running an Ansible playbook.

This procedure configures RELP on all hosts in the server group in the Ansible inventory. The RELP configuration uses TLS to encrypt the message transmission for secure transfer of logs over the network.

Prerequisites

  • You have permissions to run playbooks on managed nodes on which you want to configure RELP.
  • The managed nodes are listed in the inventory file on the control node.
  • The ansible and rhel-system-roles packages are installed on the control node.

Procedure

  1. Create a playbook.yml file with the following content: 

    ---
- name: Deploying remote input and remote_files output
  hosts: server
  roles:
    - rhel-system-roles.logging
  vars:
    logging_inputs:
      - name: relp_server
        type: relp
        port: 20514
        tls: true
        ca_cert: _/etc/pki/tls/certs/ca.pem_
        cert: _/etc/pki/tls/certs/server-cert.pem_
        private_key: _/etc/pki/tls/private/server-key.pem_
        pki_authmode: name
        permitted_clients:
          - '_*example.client.com_'
    logging_outputs:
      - name: _remote_files_output_
        type: _remote_files_
    logging_flows:
      - name: _example_flow_
        inputs: _relp_server_
        outputs: _remote_files_output_

    The playbooks uses following settings:

    • port: Port number the remote logging system is listening.

    • tls: Ensures secure transfer of logs over the network. If you do not want a secure wrapper you can set the tls variable to false. By default tls parameter is set to true while working with RELP and requires key/certificates and triplets {ca_cert, cert, private_key} and/or {ca_cert_src, cert_src, private_key_src}.

      • If {ca_cert_src, cert_src, private_key_src} triplet is set, the default locations /etc/pki/tls/certs and /etc/pki/tls/private are used as the destination on the managed node to transfer files from control node. In this case, the file names are identical to the original ones in the triplet
      • If {ca_cert, cert, private_key} triplet is set, files are expected to be on the default path before the logging configuration.
      • If both the triplets are set, files are transferred from local path from control node to specific path of the managed node.
    • ca_cert: Represents the path to CA certificate. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    • cert: Represents the path to cert. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    • private_key: Represents the path to private key. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    • ca_cert_src: Represents local CA cert file path which is copied to the target host. If ca_cert is specified, it is copied to the location.
    • cert_src: Represents the local cert file path which is copied to the target host. If cert is specified, it is copied to the location.
    • private_key_src: Represents the local key file path which is copied to the target host. If private_key is specified, it is copied to the location.
    • pki_authmode: Accepts the authentication mode as name or fingerprint.
    • permitted_clients: List of clients that will be allowed by the logging server to connect and send logs over TLS.
    • inputs: List of logging input dictionary.
    • outputs: List of logging output dictionary.

  2. Optional: Verify playbook syntax. 

    # ansible-playbook --syntax-check playbook.yml

  3. Run the playbook: 

    # ansible-playbook -i inventory_file playbook.yml

12.8. Additional resources

Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.