16.3. Setting up Grafana by using the metrics RHEL system role to monitor multiple hosts with Performance Co-Pilot

Procedure

1. Store your sensitive variables in an encrypted file:

   1. Create the vault:

      $ ansible-vault create ~/vault.yml
      New Vault password: <vault_password>
      Confirm New Vault password: <vault_password>

   2. After the ansible-vault create command opens an editor, enter the sensitive data in the <key>: <value> format:

      grafana_admin_pwd: <password>

   3. Save the changes, and close the editor. Ansible encrypts the data in the vault.

2. Create a playbook file, for example, ~/playbook.yml, with the following content:

   ---
   - name: Monitoring performance metrics
     hosts: managed-node-01.example.com
     vars_files:
       - ~/vault.yml
     tasks:
       - name: Set up Grafana to monitor multiple hosts
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.metrics
         vars:
           metrics_graph_service: true
           metrics_query_service: true
           metrics_monitored_hosts:
             - <pcp_host_1.example.com>
             - <pcp_host_2.example.com>
           metrics_manage_firewall: true
           metrics_manage_selinux: true
   
       - name: Set Grafana admin password
         ansible.builtin.shell:
           cmd: grafana-cli admin reset-admin-password "{{ grafana_admin_pwd }}"

   The settings specified in the example playbook include the following:

   metrics_graph_service: true

   Installs Grafana and the PCP plug-in. Additionally, the role adds the PCP Vector, PCP Redis, and PCP bpftrace data sources to Grafana.

   metrics_query_service: <true|false>

   Defines whether the role should install and configure Redis for centralized metric recording. If enabled, data collected from PCP clients is stored in Redis and, as a result, you can also display historical data instead of only live data.

   metrics_monitored_hosts: <list_of_hosts>

   Defines the list of hosts to monitor. In Grafana, you can then display the data of these hosts and, additionally, the host that runs Grafana.

   metrics_manage_firewall: <true|false>

   Defines whether the role should open the required ports in the firewalld service. If you set this variable to true, you can, for example, access Grafana remotely.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file on the control node.

3. Validate the playbook syntax:

   $ ansible-playbook --ask-vault-pass --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

4. Run the playbook:

   $ ansible-playbook --ask-vault-pass ~/playbook.yml

Verification

1. Open http://<grafana_server_IP_or_hostname>:3000 in your browser, and log in as the admin user with the password you set in the procedure.

2. Display monitoring data:

   • To display live data:

     1. Click Menu Apps Performance Co-Pilot PCP Vector Checklist
     2. By default, the graphs display metrics from the host that runs Grafana. To switch to a different host, enter the hostname in the hostspec field and press Enter.
   • To display historical data stored in a Redis database: Create a panel with a PCP Valkey data source. This requires that you set metrics_query_service: true in the playbook.