Chapter 10. Setting up graphical representation of PCP metrics

Procedure

1. Install the following packages:

   Copy to Clipboard Toggle word wrap

   # dnf install grafana grafana-pcp

2. Restart and enable the following service:

   Copy to Clipboard Toggle word wrap

   # systemctl restart grafana-server
   # systemctl enable grafana-server

3. Open the server’s firewall for network traffic to the Grafana service.

   Copy to Clipboard Toggle word wrap

   # firewall-cmd --permanent --add-service=grafana
   success
   
   # firewall-cmd --reload
   success

Procedure

1. On the client system, open http://<grafana_server_IP_address_or_hostname>:3000 in your browser.

2. For the first login, enter admin in both the Email or username and Password field.

   Grafana prompts to set a New password to create a secured account.

3. In the menu, navigate to Administration and then click Plugins.
4. In the Plugins tab, type performance co-pilot in the Search Grafana plugins text box and then click Performance Co-Pilot (PCP) plugin.
5. In the Plugins / Performance Co-Pilot pane, click Enable.

6. Click the Grafana icon . The Grafana Home page is displayed.

   Figure 10.1. Home Dashboard

   
   Note

   The top right corner of the screen has a settings (gear) icon that controls the general Dashboard settings.

7. In the Grafana Home page, click Add your first data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources. For more information about adding data source, see:

   • To add pcp redis data source, view default dashboard, create a panel, and an alert rule, see Creating panels and alert in PCP Redis data source.
   • To add pcp bpftrace data source and view the default dashboard, see Viewing the PCP bpftrace System Analysis dashboard.
   • To add pcp vector data source, view the default dashboard, and to view the vector checklist, see Viewing the PCP Vector Checklist.
8. Optional: From the menu, hover over the admin profile icon to update your Profile, view Notification history, Change password, or to Sign out.

Procedure

1. As a root user, open the /etc/grafana/grafana.ini file and adjust the following options in the [server] section to reflect the following:

   Copy to Clipboard Toggle word wrap

   protocol = https
   cert_key = /etc/grafana/grafana.key
   cert_file = /etc/grafana/grafana.crt

2. Ensure grafana can access the certificates:

   Copy to Clipboard Toggle word wrap

   # su grafana -s /bin/bash -c \
     'ls -1 /etc/grafana/grafana.crt /etc/grafana/grafana.key'
   /etc/grafana/grafana.crt
   /etc/grafana/grafana.key

3. Restart and enable the Grafana service to apply the configuration changes:

   Copy to Clipboard Toggle word wrap

   # systemctl restart grafana-server
   # systemctl enable grafana-server

Verification

1. On the client system, open a browser and access the Grafana server machine on port 3000, using the https://192.0.2.0:3000 link. Replace 192.0.2.0 with your machine IP.

2. Confirm the lock icon is displayed beside the address bar.

   Note

   If the protocol is set to http and an HTTPS connection is attempted, you will receive a ERR_SSL_PROTOCOL_ERROR error. If the protocol is set to https and an HTTP connection is attempted, the Grafana server responds with a “Client sent an HTTP request to an HTTPS server” message.

Procedure

1. Install the redis package:

   Copy to Clipboard Toggle word wrap

   # dnf install redis

2. Start and enable the following services:

   Copy to Clipboard Toggle word wrap

   # systemctl start pmproxy redis
   # systemctl enable pmproxy redis

3. Restart the Grafana server:

   Copy to Clipboard Toggle word wrap

   # systemctl restart grafana-server

Procedure

1. As a root user, open the /etc/redis/redis.conf file and adjust the TLS/SSL options to reflect the following properties:

   Copy to Clipboard Toggle word wrap

   port 0
   tls-port 6379
   tls-cert-file /etc/redis/redis.crt
   tls-key-file /etc/redis/redis.key
   tls-client-key-file /etc/redis/client.key
   tls-client-cert-file /etc/redis/client.crt
   tls-ca-cert-file /etc/redis/ca.crt

2. Ensure redis can access the TLS certificates:

   Copy to Clipboard Toggle word wrap

   # su redis -s /bin/bash -c \
     'ls -1 /etc/redis/ca.crt /etc/redis/redis.key /etc/redis/redis.crt'
   /etc/redis/ca.crt
   /etc/redis/redis.crt
   /etc/redis/redis.key

3. Restart the redis server to apply the configuration changes:

   Copy to Clipboard Toggle word wrap

   # systemctl restart redis

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type redis in the Filter by name or type text box and then click PCP Redis.

4. In the Data Sources / PCP Redis pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.
   2. Click Dashboards tab Import PCP Redis: Host Overview to see a dashboard with an overview of any useful metrics.

   3. Optional: In the drop-down menu next to the clock icon , you can set the timeline of the displayed metrics either by setting the absolute time range or by selecting a predefined range. You can also use the zoom out icon to modify the displayed time range.

      Note

      The time frame displayed by default might not be aligned with the time frame covered by the archive files created by the pmlogger service.

      Figure 10.2. PCP Redis: Host Overview

      

5. Add a new panel:

   1. From the plus sign drop-down menu, select New dashboard.
   2. From the Add drop-down menu, select Visualization.
   3. In the Query tab, select the pcp-redis-datasource as the Data source.
   4. In the text field below A, enter metric, for example, kernel.all.load to visualize the kernel load graph.
   5. Optional: From the Time series drop-down menu on the right, select another format of visualization, for example, Bar chart, Table, or Heatmap.
   6. Optional: Add Panel title and Description, and update other options.
   7. Click Save to apply changes and save the dashboard. Add Title.

   8. Click Apply to apply changes and go back to the dashboard.

      Figure 10.3. PCP Redis query panel

      

6. Create an alert rule:

   1. In the PCP Redis query panel, click Alert and then click New alert rule.
   2. Enter alert rule name.
   3. Define query and alert condition.
   4. Set evaluation behavior.
   5. Optional: Add annotations.
   6. Add labels and notifications.
   7. Click Save rule an exit to apply changes in alert rules.

   8. Click Apply to apply changes and go back to the dashboard.

      Figure 10.4. Creating alerts in the PCP Redis panel

      

      To add a notification channel for the created alert rule to receive an alert notification from Grafana, see Adding notification channels for alerts.

Procedure

1. Configure SMTP and add a valid sender’s email address in the /etc/grafana/grafana.ini file:

   Copy to Clipboard Toggle word wrap

   [smtp]
   enabled = true
   from_address = <sender_email_address>

2. Restart the Grafana server.

   Copy to Clipboard Toggle word wrap

   # systemctl restart grafana-server.service

3. From the menu, select Alerting Contact points + Add contact point.

   Figure 10.5. Alerting in Grafana

   

4. In the Create contact point details view, perform the following:

   1. Enter your name in the Name text box.
   2. Select the Integration type, for example, Email and enter the email address or multiple email addresses.
   3. Optional: Configure Optional Email settings and Notification settings.
5. Click Save contact point.

6. Select a notification channel in the alert rule:

   1. From the Alerting menu, select Notification policies.
   2. Click the three dots icon on the far right of Default policy and select Edit.
   3. Choose the Contact point you have just created and click Update default policy.
   4. Optional: Configure a nested policy in addition to the default policy.
   5. Optional: Configure Mute Timings.

Procedure

1. Install the sasl framework for the scram-sha-256 authentication mechanism:

   Copy to Clipboard Toggle word wrap

   # dnf install cyrus-sasl-scram cyrus-sasl-lib

2. Specify the supported authentication mechanism and the user database path in the pmcd.conf file:

   Copy to Clipboard Toggle word wrap

   # vi /etc/sasl2/pmcd.conf
   
   mech_list: scram-sha-256
   
   sasldb_path: /etc/pcp/passwd.db

3. Create a new user:

   Copy to Clipboard Toggle word wrap

   # useradd -r metrics

   Replace metrics by your user name.

4. Add the created user in the user database:

   Copy to Clipboard Toggle word wrap

   # saslpasswd2 -a pmcd metrics
   
   Password:
   Again (for verification):

   To add the created user, you are required to enter the metrics account password.

5. Set the permissions of the user database:

   Copy to Clipboard Toggle word wrap

   # chown root:pcp /etc/pcp/passwd.db
   # chmod 640 /etc/pcp/passwd.db

6. Restart the pmcd service:

   Copy to Clipboard Toggle word wrap

   # systemctl restart pmcd

Procedure

1. Install the pcp-pmda-bpftrace package:

   Copy to Clipboard Toggle word wrap

   # dnf install pcp-pmda-bpftrace

2. Edit the bpftrace.conf file and add the user that you have created in Setting up authentication between PCP components:

   Copy to Clipboard Toggle word wrap

   # vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf
   
   [dynamic_scripts]
   enabled = true
   auth_enabled = true
   allowed_users = root,metrics

   Replace metrics by your user name.

3. Install bpftrace PMDA:

   Copy to Clipboard Toggle word wrap

   # cd /var/lib/pcp/pmdas/bpftrace/
   # ./Install
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check bpftrace metrics have appeared ... 7 metrics and 6 values

   The pmda-bpftrace is now installed, and can only be used after authenticating your user. For more information, see Viewing the PCP bpftrace System Analysis dashboard.

Procedure

1. Log into the Grafana web UI.
2. In the menu, navigate to Connections Data sources + Add new data source.
3. In the Add data source pane, type bpftrace in the Filter by name or type text box and then click PCP bpftrace.

4. In the Data Sources / pcp-bpftrace-datasource pane, perform the following:

   1. Add http://localhost:44322 in the URL field.
   2. Toggle the Basic Auth option and add the created user credentials in the User and Password field.

   3. Click Save & Test.

      Figure 10.6. Adding PCP bpftrace in the data source

      

   4. Click Dashboards tab Import PCP bpftrace: System Analysis to see a dashboard with an overview of any useful metrics.

      Figure 10.7. PCP bpftrace: System Analysis

      

Procedure

1. Install the pcp-pmda-bcc package:

   Copy to Clipboard Toggle word wrap

   # dnf install pcp-pmda-bcc

2. Install the bcc PMDA:

   Copy to Clipboard Toggle word wrap

   # cd /var/lib/pcp/pmdas/bcc
   # ./Install
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state.
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: Enabled modules:
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork',
   [...]
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values

Procedure

1. Log into the Grafana web UI.
2. In the menu, navigate to Connections Data sources + Add new data source.
3. In the Add data source pane, type vector in the Filter by name or type text box and then click PCP Vector.

4. In the Data Sources / pcp-vector-datasource pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab Import PCP Vector: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 10.8. PCP Vector: Host Overview

      

5. In the menu, navigate to Apps Performance Co-Pilot PCP Vector Checklist.

   In the PCP checklist, click the question mark icon for help or the warning icon to view the related troubleshooting or reference links.

   Figure 10.9. Performance Co-Pilot / PCP Vector Checklist

   

Procedure

1. In the menu, select Dashboards New Dashboard + Add visualization.
2. In the Select data source pane, select pcp-redis-datasource.
3. In the Query tab, in the text field below A, enter a metric, for example, kernel.all.load to visualize the kernel load graph.
4. From the Time series drop-down menu on the right, select Heatmap.
5. Optional: In the Panel Options dropdown menu, add a Panel Title and Description.

6. In the Heatmap dropdown menu, under the Calculate from data setting, click Yes.

   Heatmap

   

7. Optional: In the Colors dropdown menu, change the Scheme from the default Orange and select the number of steps (color shades).

8. Optional: In the Tooltip dropdown menu, click the toggle to display a cell’s position within its specific histogram when hovering your cursor over a cell in the heatmap. For example:

   Show histogram (Y Axis) cell display