Chapter 16. Configuring performance monitoring with PCP by using RHEL system roles

Procedure

1. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Monitoring performance metrics
     hosts: managed-node-01.example.com
     tasks:
       - name: Configure Performance Co-Pilot
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.metrics
         vars:
           metrics_retention_days: 14
           metrics_manage_firewall: true
           metrics_manage_selinux: true

   Copy to Clipboard Toggle word wrap

   The settings specified in the example playbook include the following:

   metrics_retention_days: <number>

   Sets the number of days after which the pmlogger_daily systemd timer removes old PCP archives.

   metrics_manage_firewall: <true|false>

   Defines whether the role should open the required ports in the firewalld service. If you want to remotely access PCP on the managed nodes, set this variable to true.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Copy to Clipboard Toggle word wrap

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

   Copy to Clipboard Toggle word wrap

Procedure

1. Store your sensitive variables in an encrypted file:

   1. Create the vault:

      $ ansible-vault create ~/vault.yml
      New Vault password: <vault_password>
      Confirm New Vault password: <vault_password>

      Copy to Clipboard Toggle word wrap

   2. After the ansible-vault create command opens an editor, enter the sensitive data in the <key>: <value> format:

      metrics_usr: <username>
      metrics_pwd: <password>

      Copy to Clipboard Toggle word wrap
   3. Save the changes, and close the editor. Ansible encrypts the data in the vault.

2. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Monitoring performance metrics
     hosts: managed-node-01.example.com
     vars_files:
       - ~/vault.yml
     tasks:
       - name: Configure Performance Co-Pilot
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.metrics
         vars:
           metrics_retention_days: 14
           metrics_manage_firewall: true
           metrics_manage_selinux: true
           metrics_username: "{{ metrics_usr }}"
           metrics_password: "{{ metrics_pwd }}"

   Copy to Clipboard Toggle word wrap

   The settings specified in the example playbook include the following:

   metrics_retention_days: <number>

   Sets the number of days after which the pmlogger_daily systemd timer removes old PCP archives.

   metrics_manage_firewall: <true|false>

   Defines whether the role should open the required ports in the firewalld service. If you want to remotely access PCP on the managed nodes, set this variable to true.

   metrics_username: <username>

   The role creates this user locally on the managed node, adds the credentials to the /etc/pcp/passwd.db Simple Authentication and Security Layer (SASL) database, and configures authentication in PCP. Additionally, if you set metrics_from_bpftrace: true in the playbook, PCP uses this account to register bpftrace scripts.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file on the control node.

3. Validate the playbook syntax:

   $ ansible-playbook --ask-vault-pass --syntax-check ~/playbook.yml

   Copy to Clipboard Toggle word wrap

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

4. Run the playbook:

   $ ansible-playbook --ask-vault-pass ~/playbook.yml

   Copy to Clipboard Toggle word wrap

Procedure

1. Store your sensitive variables in an encrypted file:

   1. Create the vault:

      $ ansible-vault create ~/vault.yml
      New Vault password: <vault_password>
      Confirm New Vault password: <vault_password>

      Copy to Clipboard Toggle word wrap

   2. After the ansible-vault create command opens an editor, enter the sensitive data in the <key>: <value> format:

      grafana_admin_pwd: <password>

      Copy to Clipboard Toggle word wrap
   3. Save the changes, and close the editor. Ansible encrypts the data in the vault.

2. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Monitoring performance metrics
     hosts: managed-node-01.example.com
     vars_files:
       - ~/vault.yml
     tasks:
       - name: Set up Grafana to monitor multiple hosts
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.metrics
         vars:
           metrics_graph_service: true
           metrics_query_service: true
           metrics_monitored_hosts:
             - <pcp_host_1.example.com>
             - <pcp_host_2.example.com>
           metrics_manage_firewall: true
           metrics_manage_selinux: true
   
       - name: Set Grafana admin password
         ansible.builtin.shell:
           cmd: grafana-cli admin reset-admin-password "{{ grafana_admin_pwd }}"

   Copy to Clipboard Toggle word wrap

   The settings specified in the example playbook include the following:

   metrics_graph_service: true

   Installs Grafana and the PCP plug-in. Additionally, the role adds the PCP Vector, PCP Redis, and PCP bpftrace data sources to Grafana.

   metrics_query_service: <true|false>

   Defines whether the role should install and configure Redis for centralized metric recording. If enabled, data collected from PCP clients is stored in Redis and, as a result, you can also display historical data instead of only live data.

   metrics_monitored_hosts: <list_of_hosts>

   Defines the list of hosts to monitor. In Grafana, you can then display the data of these hosts and, additionally, the host that runs Grafana.

   metrics_manage_firewall: <true|false>

   Defines whether the role should open the required ports in the firewalld service. If you set this variable to true, you can, for example, access Grafana remotely.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file on the control node.

3. Validate the playbook syntax:

   $ ansible-playbook --ask-vault-pass --syntax-check ~/playbook.yml

   Copy to Clipboard Toggle word wrap

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

4. Run the playbook:

   $ ansible-playbook --ask-vault-pass ~/playbook.yml

   Copy to Clipboard Toggle word wrap

Verification

1. Open http://<grafana_server_IP_or_hostname>:3000 in your browser, and log in as the admin user with the password you set in the procedure.

2. Display monitoring data:

   • To display live data:

     1. Click the Performance Co-Pilot icon in the navigation bar on the left, and select PCP Vector Checklist.
     2. By default, the graphs display metrics from the host that runs Grafana. To switch to a different host, enter the hostname in the hostspec field and press Enter.
   • To display historical data stored in a Redis database: Create a panel with a PCP Redis data source. This requires that you set metrics_query_service: true in the playbook.