Chapter 8. Setting up graphical representation of PCP metrics

Procedure

1. Install the following packages:

   # dnf install grafana grafana-pcp

2. Restart and enable grafana-server:

   # systemctl restart grafana-server

   # systemctl enable grafana-server

3. Open the server’s firewall for network traffic to the Grafana service.

   # firewall-cmd --permanent --add-service=grafana

   success

   # firewall-cmd --reload

   success

Procedure

1. Install the valkey package:

   # dnf install valkey

2. Start and enable the pmproxy and valkey services:

   # systemctl start pmproxy valkey

   # systemctl enable pmproxy valkey

3. Restart grafana-server:

   # systemctl restart grafana-server

Procedure

1. On the client system, open a browser and access the grafana-server on port 3000 by using the http://192.0.2.0:3000 link.

   Replace 192.0.2.0 with your machine IP when accessing Grafana web UI from a remote machine, or with localhost when accessing the web UI locally.

2. For the first login, enter admin in both the Email or username and Password fields.
3. Grafana prompts to set a New password to create a secured account. If you want to set it later, click Skip.
4. From the more options icon (☰) on the top left, click Administration > Plugins.
5. In the Plugins tab, type performance co-pilot in the Search by name or type text box and then click the Performance Co-Pilot (PCP) plugin.
6. In the Plugins / Performance Co-Pilot pane, click Enable.

7. Click the Grafana icon. The Grafana Home page is displayed.

   Figure 8.1. Home Dashboard

   
   Note

   The top corner of the screen has a Settings icon, but it controls the general Dashboard settings.

8. In the Grafana Home page, click Add your first data source to add Valkey, PCP bpftrace, and PCP Vector data sources.

   • To add PCP valkey data source, view default dashboard, create a panel, and an alert rule, see Creating panels and alerts in PCP valkey data source.
   • To add pcp bpftrace data source and view the default dashboard, see Viewing the PCP bpftrace System Analysis dashboard.
   • To add pcp vector data source, view the default dashboard, and to view the vector checklist, see Viewing the PCP Vector Checklist.

9. Optional: From the menu, hover over the admin profile icon to change the Preferences including Edit Profile, Change Password, or to Sign out.

   For more information, see the grafana-cli and grafana-server man pages on your system.

Procedure

1. As a root user, open the /etc/valkey/valkey.conf file and adjust the TLS/SSL options to reflect the following properties:

   port 0
   tls-port 6379
   tls-cert-file /etc/valkey/valkey.crt
   tls-key-file /etc/valkey/valkey.key
   tls-client-key-file /etc/valkey/client.key
   tls-client-cert-file /etc/valkey/client.crt
   tls-ca-cert-file /etc/valkey/ca.crt

2. Ensure valkey can access the TLS certificates:

   # su valkey -s /bin/bash -c \

     'ls -1 /etc/valkey/ca.crt /etc/valkey/valkey.key /etc/valkey/valkey.crt'
   /etc/valkey/ca.crt
   /etc/valkey/valkey.crt
   /etc/valkey/valkey.key

3. Restart the valkey server to apply the configuration changes:

   # systemctl restart valkey

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type valkey in the Filter by name or type text box and then click Valkey.

4. In the Data Sources / Valkey pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab > Import > Valkey: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 8.2. Valkey: Host overview

      

5. Add a new panel:

   1. From the menu, hover over the Create icon > Dashboard > Add new panel icon to add a panel.
   2. In the Query tab, select the Valkey from the query list instead of the selected default option and in the text field of A, enter metric, for example, kernel.all.load to visualize the kernel load graph.
   3. Optional: Add Panel title and Description, and update other options from the Settings.
   4. Click Save to apply changes and save the dashboard. Add Dashboard name.

   5. Click Apply to apply changes and go back to the dashboard.

      Figure 8.3. Valkey query panel

      

6. Create an alert rule:

   1. In the Valkey query panel, click Alert and then click Create Alert.
   2. Edit the Name, Evaluate query, and For fields from the Rule, and specify the Conditions for your alert.

   3. Click Save to apply changes and save the dashboard. Click Apply to apply changes and go back to the dashboard.

      Figure 8.4. Creating alerts in the Valkey panel

      
   4. Optional: In the same panel, scroll down and click Delete icon to delete the created rule.
   5. Optional: From the menu, click Alerting icon to view the created alert rules with different alert statuses, to edit the alert rule, or to pause the existing rule from the Alert Rules tab. To add a notification channel for the created alert rule to receive an alert notification from Grafana, see Adding notification channels for alerts.

Procedure

1. From the menu, hover over the Alerting icon > click Notification channels > Add channel.

2. In the Add notification channel details pane, perform the following:

   1. Enter your name in the Name text box.
   2. Select the communication Type, for example, Email and enter the email address. You can add multiple email addresses using the ; separator.
   3. Optional: Configure Optional Email settings and Notification settings.
3. Click Save.

4. Select a notification channel in the alert rule:

   1. From the menu, hover over the Alerting icon and then click Alert rules.
   2. From the Alert Rules tab, click the created alert rule.
   3. On the Notifications tab, select your notification channel name from the Send to option, and then add an alert message.
5. Click Apply.

Procedure

1. Install the SASL framework for the scram-sha-256 authentication mechanism:

   # dnf install cyrus-sasl-scram cyrus-sasl-lib

2. Specify the supported authentication mechanism and the user database path in the pmcd.conf file:

   # vi /etc/sasl2/pmcd.conf

   mech_list: scram-sha-256
   sasldb_path: /etc/pcp/passwd.db

3. Create a new user:

   # useradd -r metrics

   Replace metrics by your user name.

4. Add the created user in the user database:

   # saslpasswd2 -a pmcd metrics

   Password:
   Again (for verification):

   To add the created user, it is required to enter the metrics account password.

5. Set the permissions of the user database:

   # chown root:pcp /etc/pcp/passwd.db

   # chmod 640 /etc/pcp/passwd.db

6. Restart the pmcd service:

   # systemctl restart pmcd

Procedure

1. Install the pcp-pmda-bpftrace package:

   # dnf install pcp-pmda-bpftrace

2. Edit the bpftrace.conf file and add the user:

   # vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf

   [dynamic_scripts]
   enabled = true
   auth_enabled = true
   allowed_users = root,metrics

   Replace metrics by your user name from Setting up authentication between PCP components.

3. Install bpftrace PMDA:

   # cd /var/lib/pcp/pmdas/bpftrace/

   # ./Install

   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD …
   Check bpftrace metrics have appeared ... 7 metrics and 6 values

   The pmda-bpftrace is now installed, and can only be used after authenticating your user. For more information, see Viewing the PCP bpftrace System Analysis dashboard.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type bpftrace in the Filter by name or type text box and then click PCP bpftrace.

4. In the Data Sources / PCP bpftrace pane, perform the following:

   1. Add http://localhost:44322 in the URL field.
   2. Toggle the Basic Auth option and add the created user credentials in the User and Password field.

   3. Click Save & Test.

      Figure 8.5. Adding PCP bpftrace in the data source

      

   4. Click Dashboards tab > Import > PCP bpftrace: System Analysis to see a dashboard with an overview of any useful metrics.

      Figure 8.6. PCP bpftrace: System Analysis

      

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type vector in the Filter by name or type text box and then click PCP Vector.

4. In the Data Sources / PCP Vector pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab > Import * PCP Vector: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 8.7. PCP Vector: Host Overview

      

5. From the menu, hover over the Performance Co-Pilot plugin and then click PCP Vector Checklist.

   In the PCP checklist, click help or warning icon to view the related troubleshooting or reference links.

   Figure 8.8. Performance Co-Pilot / PCP Vector Checklist

   

Procedure

1. Hover the cursor over the Dashboards tab and click + New dashboard.
2. In the Add panel menu, click Add a new panel.

3. In the Query tab:

   1. Select Valkey from the query list instead of the selected default option.
   2. In the text field of A, enter a metric, for example, kernel.all.load to visualize the kernel load graph.
4. Click the visualization dropdown menu, which is set to Time series by default, and then click Heatmap.
5. Optional: In the Panel Options dropdown menu, add a Panel Title and Description.

6. In the Heatmap dropdown menu, under the Calculate from data setting, click Yes.

   Figure 8.9. Heatmap

   
7. Optional: In the Colors dropdown menu, change the Scheme from the default Orange and select the number of steps (color shades).

8. Optional: In the Tooltip dropdown menu, under the Show histogram (Y Axis) setting, click the toggle to display a cell’s position within its specific histogram when hovering your cursor over a cell in the heatmap. For example:

   

Procedure

1. View all Grafana-related SELinux booleans:

   # semanage boolean -l | grep grafana

2. Enable a SELinux boolean:

   # setsebool [-P] <boolean_name> on

   • Replace <boolean_name> with the name of the SELinux boolean you want to enable.
   • Use the -P option to make the change persistent across system reboots.

3. Optional: Turn off any SELinux option:

   # setsebool [-P] <boolean_name> off