Monitoring and managing system status and performance

Procedure

1. Edit the /etc/tuned/cpu-partitioning-variables.conf file with the following changes:

   1. Comment the isolated_cores=${f:calc_isolated_cores:1} line:

      # isolated_cores=${f:calc_isolated_cores:1}

   2. Add the following information for isolated CPUS:

      # All isolated CPUs:
      isolated_cores=2-23
      # Isolated CPUs without the kernel’s scheduler load balancing:
      no_balance_cores=2,3

2. Set the cpu-partitioning TuneD profile:

   # tuned-adm profile cpu-partitioning

3. Reboot the system.

   After rebooting, the system is tuned for low-latency, according to the isolation in the cpu-partitioning figure. The application can use taskset to pin the reader and writer threads to CPUs 2 and 3, and the remaining application threads on CPUs 4-23.

Procedure

1. Create the /etc/tuned/my_profile directory:

   # mkdir /etc/tuned/my_profile

2. Create a tuned.conf file in this directory, and add the following content:

   # vi /etc/tuned/my_profile/tuned.conf
   [main]
   summary=Customized tuning on top of cpu-partitioning
   include=cpu-partitioning
   [cpu]
   force_latency=cstate.id:0|1

3. Use the new profile:

   # tuned-adm profile my_profile

Procedure

1. Install the TuneD package:

   # yum install tuned

2. Enable and start the TuneD service:

   # systemctl enable --now tuned

3. Optional: Install TuneD profiles for real-time systems:

   For the TuneD profiles for real-time systems enable rhel-8 repository.

   # subscription-manager repos --enable=rhel-8-for-x86_64-nfv-beta-rpms

   Install it.

   # yum install tuned-profiles-realtime tuned-profiles-nfv

4. Verify that a TuneD profile is active and applied:

   $ tuned-adm active
   
   Current active profile: throughput-performance

   Note

   The active profile TuneD automatically presets differs based on your machine type and system settings.

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

Procedure

1. Optional: You can let TuneD recommend the most suitable profile for your system:

   # tuned-adm recommend
   
   throughput-performance

2. Activate a profile:

   # tuned-adm profile selected-profile

   Alternatively, you can activate a combination of multiple profiles:

   # tuned-adm profile selected-profile1 selected-profile2

   Example 2.3. A virtual machine optimized for low power consumption

   The following example optimizes the system to run in a virtual machine with the best performance and concurrently tunes it for low power consumption, while the low power consumption is the priority:

   # tuned-adm profile virtual-guest powersave

3. View the current active TuneD profile on your system:

   # tuned-adm active
   
   Current active profile: selected-profile

4. Reboot the system:

   # reboot

Procedure

1. In the /etc/tuned/ directory, create a new directory named the same as the profile that you want to create:

   # mkdir /etc/tuned/my-profile

2. In the new directory, create a file named tuned.conf. Add a [main] section and plug-in definitions in it, according to your requirements.

   For example, see the configuration of the balanced profile:

   [main]
   summary=General non-specialized TuneD profile
   
   [cpu]
   governor=conservative
   energy_perf_bias=normal
   
   [audio]
   timeout=10
   
   [video]
   radeon_powersave=dpm-balanced, auto
   
   [scsi_host]
   alpm=medium_power

3. To activate the profile, use:

   # tuned-adm profile my-profile

4. Verify that the TuneD profile is active and the system settings are applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

Procedure

1. In the /etc/tuned/ directory, create a new directory named the same as the profile that you want to create:

   # mkdir /etc/tuned/modified-profile

2. In the new directory, create a file named tuned.conf, and set the [main] section as follows:

   [main]
   include=parent-profile

   Replace parent-profile with the name of the profile you are modifying.

3. Include your profile modifications.

   Example 3.18. Lowering swappiness in the throughput-performance profile

   To use the settings from the throughput-performance profile and change the value of vm.swappiness to 5, instead of the default 10, use:

   [main]
   include=throughput-performance
   
   [sysctl]
   vm.swappiness=5

4. To activate the profile, use:

   # tuned-adm profile modified-profile

5. Verify that the TuneD profile is active and the system settings are applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

Procedure

1. Optional: Select an existing TuneD profile on which your profile will be based. For a list of available profiles, see TuneD profiles distributed with RHEL.

   To see which profile is currently active, use:

   $ tuned-adm active

2. Create a new directory to hold your TuneD profile:

   # mkdir /etc/tuned/my-profile

3. Find the system unique identifier of the selected block device:

   $ udevadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)'
   
   ID_WWN=0x5002538d00000000_
   ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   ID_SERIAL_SHORT=20120501030900000

   Note

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

4. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following options:

   1. Optional: Include an existing profile:

      [main]
      include=existing-profile

   2. Set the selected disk scheduler for the device that matches the WWN identifier:

      [disk]
      devices_udev_regex=IDNAME=device system unique id
      elevator=selected-scheduler

      Here:

      • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).

      • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

        To match multiple devices in the devices_udev_regex option, enclose the identifiers in parentheses and separate them with vertical bars:

        devices_udev_regex=(ID_WWN=0x5002538d00000000)|(ID_WWN=0x1234567800000000)

5. Enable your profile:

   # tuned-adm profile my-profile

Verification

1. Verify that the TuneD profile is active and applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See TuneD log file ('/var/log/tuned/tuned.log') for details.

2. Read the contents of the /sys/block/device/queue/scheduler file:

   # cat /sys/block/device/queue/scheduler
   
   [mq-deadline] kyber bfq none

   In the file name, replace device with the block device name, for example sdc.

   The active scheduler is listed in square brackets ([]).

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Monitoring performance metrics
     hosts: managed-node-01.example.com
     tasks:
       - name: Configure Performance Co-Pilot
         ansible.builtin.include_role:
           name: rhel-system-roles.metrics
         vars:
           metrics_retention_days: 14
           metrics_manage_firewall: true
           metrics_manage_selinux: true

   The settings specified in the example playbook include the following:

   metrics_retention_days: <number>

   Sets the number of days after which the pmlogger_daily systemd timer removes old PCP archives.

   metrics_manage_firewall: <true|false>

   Defines whether the role should open the required ports in the firewalld service. If you want to remotely access PCP on the managed nodes, set this variable to true.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file on the control node.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Procedure

1. Store your sensitive variables in an encrypted file:

   1. Create the vault:

      $ ansible-vault create vault.yml
      New Vault password: <vault_password>
      Confirm New Vault password: <vault_password>

   2. After the ansible-vault create command opens an editor, enter the sensitive data in the <key>: <value> format:

      metrics_usr: <username>
      metrics_pwd: <password>

   3. Save the changes, and close the editor. Ansible encrypts the data in the vault.

2. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Monitoring performance metrics
     hosts: managed-node-01.example.com
     vars_files:
       - vault.yml
     tasks:
       - name: Configure Performance Co-Pilot
         ansible.builtin.include_role:
           name: rhel-system-roles.metrics
         vars:
           metrics_retention_days: 14
           metrics_manage_firewall: true
           metrics_manage_selinux: true
   	metrics_username: "{{ metrics_usr }}"
           metrics_password: "{{ metrics_pwd }}"

   The settings specified in the example playbook include the following:

   metrics_retention_days: <number>

   Sets the number of days after which the pmlogger_daily systemd timer removes old PCP archives.

   metrics_manage_firewall: <true|false>

   Defines whether the role should open the required ports in the firewalld service. If you want to remotely access PCP on the managed nodes, set this variable to true.

   metrics_username: <username>

   The role creates this user locally on the managed node, adds the credentials to the /etc/pcp/passwd.db Simple Authentication and Security Layer (SASL) database, and configures authentication in PCP. Additionally, if you set metrics_from_bpftrace: true in the playbook, PCP uses this account to register bpftrace scripts.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file on the control node.

3. Validate the playbook syntax:

   $ ansible-playbook --ask-vault-pass --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

4. Run the playbook:

   $ ansible-playbook --ask-vault-pass ~/playbook.yml

Procedure

1. Store your sensitive variables in an encrypted file:

   1. Create the vault:

      $ ansible-vault create vault.yml
      New Vault password: <vault_password>
      Confirm New Vault password: <vault_password>

   2. After the ansible-vault create command opens an editor, enter the sensitive data in the <key>: <value> format:

      grafana_admin_pwd: <password>

   3. Save the changes, and close the editor. Ansible encrypts the data in the vault.

2. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Monitoring performance metrics
     hosts: managed-node-01.example.com
     vars_files:
       - vault.yml
     tasks:
       - name: Set up Grafana to monitor multiple hosts
         ansible.builtin.include_role:
           name: rhel-system-roles.metrics
         vars:
           metrics_graph_service: true
           metrics_query_service: true
           metrics_monitored_hosts:
             - <pcp_host_1.example.com>
             - <pcp_host_2.example.com>
           metrics_manage_firewall: true
           metrics_manage_selinux: true
   
       - name: Set Grafana admin password
         ansible.builtin.shell:
           cmd: grafana-cli admin reset-admin-password "{{ grafana_admin_pwd }}"

   The settings specified in the example playbook include the following:

   metrics_graph_service: true

   Installs Grafana and the PCP plug-in. Additionally, the role adds the PCP Vector, PCP Redis, and PCP bpftrace data sources to Grafana.

   metrics_query_service: <true|false>

   Defines whether the role should install and configure Redis for centralized metric recording. If enabled, data collected from PCP clients is stored in Redis and, as a result, you can also display historical data instead of only live data.

   metrics_monitored_hosts: <list_of_hosts>

   Defines the list of hosts to monitor. In Grafana, you can then display the data of these hosts and, additionally, the host that runs Grafana.

   metrics_manage_firewall: <true|false>

   Defines whether the role should open the required ports in the firewalld service. If you set this variable to true, you can, for example, access Grafana remotely.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file on the control node.

3. Validate the playbook syntax:

   $ ansible-playbook --ask-vault-pass --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

4. Run the playbook:

   $ ansible-playbook --ask-vault-pass ~/playbook.yml

Verification

1. Open http://<grafana_server_IP_or_hostname>:3000 in your browser, and log in as the admin user with the password you set in the procedure.

2. Display monitoring data:

   • To display live data:

     1. Click the Performance Co-Pilot icon in the navigation bar on the left, and select PCP Vector Checklist.
     2. By default, the graphs display metrics from the host that runs Grafana. To switch to a different host, enter the hostname in the hostspec field and press Enter.
   • To display historical data stored in a Redis database: Create a panel with a PCP Redis data source. This requires that you set metrics_query_service: true in the playbook.

Procedure

1. Install the pcp package:

   # yum install pcp

2. Enable and start the pmcd service on the host machine:

   # systemctl enable pmcd
   
   # systemctl start pmcd

Procedure

1. Update the pmlogger configuration:

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

2. Start the pmcd and pmlogger services:

   # systemctl start pmcd.service
   
   # systemctl start pmlogger.service

3. Execute the required operations to record the performance data.

4. Stop the pmcd and pmlogger services:

   # systemctl stop pmcd.service
   
   # systemctl stop pmlogger.service

5. Save the output and save it to a tar.gz file named based on the host name and the current date and time:

   # cd /var/log/pcp/pmlogger/
   
   # tar -czf $(hostname).$(date +%F-%Hh%M).pcp.tar.gz $(hostname)

   Extract this file and analyze the data using PCP tools.

Prerequisites

1. Install the pcp-pmda-redis package:

   # yum install pcp-pmda-redis

2. Install the redis PMDA:

   # cd /var/lib/pcp/pmdas/redis && ./Install

Procedure

1. Create or modify the pmlogger configuration file:

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

2. Follow pmlogconf prompts to enable or disable groups of related performance metrics and to control the logging interval for each enabled group.

Procedure

1. Install the pcp-system-tools package:

   # yum install pcp-system-tools

2. Configure an IP address for pmcd:

   # echo "-i 192.168.4.62" >>/etc/pcp/pmcd/pmcd.options

   Replace 192.168.4.62 with the IP address, the client should listen on.

   By default, pmcd is listening on the localhost.

3. Configure the firewall to add the public zone permanently:

   # firewall-cmd --permanent --zone=public --add-port=44321/tcp
   success
   
   # firewall-cmd --reload
   success

4. Set an SELinux boolean:

   # setsebool -P pcp_bind_all_unreserved_ports on

5. Enable the pmcd and pmlogger services:

   # systemctl enable pmcd pmlogger
   # systemctl restart pmcd pmlogger

Procedure

1. Install the pcp-system-tools package:

   # yum install pcp-system-tools

2. Create the /etc/pcp/pmlogger/control.d/remote file with the following content:

   # DO NOT REMOVE OR EDIT THE FOLLOWING LINE
   $version=1.1
   
   192.168.4.13 n n PCP_ARCHIVE_DIR/rhel7u4a -r -T24h10m -c config.rhel7u4a
   192.168.4.14 n n PCP_ARCHIVE_DIR/rhel6u10a -r -T24h10m -c config.rhel6u10a
   192.168.4.62 n n PCP_ARCHIVE_DIR/rhel8u1a -r -T24h10m -c config.rhel8u1a

   Replace 192.168.4.13, 192.168.4.14 and 192.168.4.62 with the client IP addresses.

   Note

   In Red Hat Enterpirse Linux 8.0, 8.1 and 8.2 use the following format for remote hosts in the control file: PCP_LOG_DIR/pmlogger/host_name.

3. Enable the pmcd and pmlogger services:

   # systemctl enable pmcd pmlogger
   # systemctl restart pmcd pmlogger

Procedure

1. Install the following packages:

   1. Install the pcp-system-tools:

      # yum install pcp-system-tools

   2. Install the pmda-postfix package to monitor postfix:

      # yum install pcp-pmda-postfix postfix

   3. Install the logging daemon:

      # yum install rsyslog

   4. Install the mail client for testing:

      # yum install mutt

2. Enable the postfix and rsyslog services:

   # systemctl enable postfix rsyslog
   # systemctl restart postfix rsyslog

3. Enable the SELinux boolean, so that pmda-postfix can access the required log files:

   # setsebool -P pcp_read_generic_logs=on

4. Install the PMDA:

   # cd /var/lib/pcp/pmdas/postfix/
   
   # ./Install
   
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Waiting for pmcd to terminate ...
   Starting pmcd ...
   Check postfix metrics have appeared ... 7 metrics and 58 values

Procedure

1. Launch the PCP Charts application from the command line:

   # pmchart

   Figure 8.1. PCP Charts application

   

   The pmtime server settings are located at the bottom. The start and pause button allows you to control:

   • The interval in which PCP polls the metric data
   • The date and time for the metrics of historical data
2. Click File and then New Chart to select metric from both the local machine and remote machines by specifying their host name or address. Advanced configuration options include the ability to manually set the axis values for the chart, and to manually choose the color of the plots.

3. Record the views created in the PCP Charts application:

   Following are the options to take images or record the views created in the PCP Charts application:

   • Click File and then Export to save an image of the current view.
   • Click Record and then Start to start a recording. Click Record and then Stop to stop the recording. After stopping the recording, the recorded metrics are archived to be viewed later.

4. Optional: In the PCP Charts application, the main configuration file, known as the view, allows the metadata associated with one or more charts to be saved. This metadata describes all chart aspects, including the metrics used and the chart columns. Save the custom view configuration by clicking File and then Save View, and load the view configuration later.

   The following example of the PCP Charts application view configuration file describes a stacking chart graph showing the total number of bytes read and written to the given XFS file system loop1:

   #kmchart
   version 1
   
   chart title "Filesystem Throughput /loop1" style stacking antialiasing off
       plot legend "Read rate"   metric xfs.read_bytes   instance  "loop1"
       plot legend "Write rate"  metric xfs.write_bytes  instance  "loop1"

Procedure

1. Install PCP:

   # yum install pcp-zeroconf

2. Install packages required for the pyodbc driver:

   # yum install gcc-c++ python3-devel unixODBC-devel
   
   # yum install python3-pyodbc

3. Install the mssql agent:

   1. Install the Microsoft SQL Server domain agent for PCP:

      # yum install pcp-pmda-mssql

   2. Edit the /etc/pcp/mssql/mssql.conf file to configure the SQL server account’s username and password for the mssql agent. Ensure that the account you configure has access rights to performance data.

      username: user_name
      password: user_password

      Replace user_name with the SQL Server account and user_password with the SQL Server user password for this account.

4. Install the agent:

   # cd /var/lib/pcp/pmdas/mssql
   # ./Install
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check mssql metrics have appeared ... 168 metrics and 598 values
   [...]

Procedure

1. Navigate to the xfs directory:

   # cd /var/lib/pcp/pmdas/xfs/

2. Install the XFS PMDA manually:

   xfs]# ./Install
   
   You will need to choose an appropriate configuration for install of
   the “xfs” Performance Metrics Domain Agent (PMDA).
   
     collector     collect performance statistics on this system
     monitor       allow this system to monitor local and/or remote systems
     both          collector and monitor configuration for this system
   
   Please enter c(ollector) or m(onitor) or (both) [b]
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Waiting for pmcd to terminate ...
   Starting pmcd ...
   Check xfs metrics have appeared ... 149 metrics and 149 values

3. Select the intended PMDA role by entering c for collector, m for monitor, or b for both. The PMDA installation script prompts you to specify one of the following PMDA roles:

   • The collector role allows the collection of performance metrics on the current system

   • The monitor role allows the system to monitor local systems, remote systems, or both

     The default option is both collector and monitor, which allows the XFS PMDA to operate correctly in most scenarios.

Procedure

1. Display the value of a metric:

   $ pminfo -f xfs.write
   
   xfs.write
       value 325262

2. Reset all the XFS metrics:

   # pmstore xfs.control.reset 1
   
   xfs.control.reset old value=0 new value=1

Procedure

1. Install the following packages:

   # yum install grafana grafana-pcp

2. Restart and enable the following service:

   # systemctl restart grafana-server
   # systemctl enable grafana-server

3. Open the server’s firewall for network traffic to the Grafana service.

   # firewall-cmd --permanent --add-service=grafana
   success
   
   # firewall-cmd --reload
   success

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

1. On the client system, open a browser and access the grafana-server on port 3000, using http://192.0.2.0:3000 link.

   Replace 192.0.2.0 with your machine IP when accessing Grafana web UI from a remote machine, or with localhost when accessing the web UI locally.

2. For the first login, enter admin in both the Email or username and Password field.

   Grafana prompts to set a New password to create a secured account. If you want to set it later, click Skip.

3. From the menu, hover over the       Configuration icon and then click Plugins.
4. In the Plugins tab, type performance co-pilot in the Search by name or type text box and then click Performance Co-Pilot (PCP) plugin.
5. In the Plugins / Performance Co-Pilot pane, click Enable.

6. Click Grafana       icon. The Grafana Home page is displayed.

   Figure 10.1. Home Dashboard

   
   Note

   The top corner of the screen has a similar       icon, but it controls the general Dashboard settings.

7. In the Grafana Home page, click Add your first data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources. For more information about adding data source, see:

   • To add pcp redis data source, view default dashboard, create a panel, and an alert rule, see Creating panels and alert in PCP Redis data source.
   • To add pcp bpftrace data source and view the default dashboard, see Viewing the PCP bpftrace System Analysis dashboard.
   • To add pcp vector data source, view the default dashboard, and to view the vector checklist, see Viewing the PCP Vector Checklist.
8. Optional: From the menu, hover over the admin profile       icon to change the Preferences including Edit Profile, Change Password, or to Sign out.

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.
3. Mail transfer agent, for example, sendmail or postfix is installed and configured.

Procedure

1. Install the redis package:

   # yum module install redis:6

   Note

   From Red Hat Enterprise Linux 8.4, Redis 6 is supported but the yum update command does not update Redis 5 to Redis 6. To update from Redis 5 to Redis 6, run:

   # yum module switch-to redis:6

2. Start and enable the following services:

   # systemctl start pmproxy redis
   # systemctl enable pmproxy redis

3. Restart the grafana-server:

   # systemctl restart grafana-server

Prerequisites

1. The PCP Redis is configured. For more information, see Configuring PCP Redis.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type redis in the Filter by name or type text box and then click PCP Redis.

4. In the Data Sources / PCP Redis pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab → Import → PCP Redis: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 10.2. PCP Redis: Host Overview

      

5. Add a new panel:

   1. From the menu, hover over the       Create icon → Dashboard → Add new panel icon to add a panel.
   2. In the Query tab, select the PCP Redis from the query list instead of the selected default option and in the text field of A, enter metric, for example, kernel.all.load to visualize the kernel load graph.
   3. Optional: Add Panel title and Description, and update other options from the Settings.
   4. Click Save to apply changes and save the dashboard. Add Dashboard name.

   5. Click Apply to apply changes and go back to the dashboard.

      Figure 10.3. PCP Redis query panel

      

6. Create an alert rule:

   1. In the PCP Redis query panel, click       Alert and then click Create Alert.
   2. Edit the Name, Evaluate query, and For fields from the Rule, and specify the Conditions for your alert.

   3. Click Save to apply changes and save the dashboard. Click Apply to apply changes and go back to the dashboard.

      Figure 10.4. Creating alerts in the PCP Redis panel

      
   4. Optional: In the same panel, scroll down and click Delete icon to delete the created rule.

   5. Optional: From the menu, click       Alerting icon to view the created alert rules with different alert statuses, to edit the alert rule, or to pause the existing rule from the Alert Rules tab.

      To add a notification channel for the created alert rule to receive an alert notification from Grafana, see Adding notification channels for alerts.

Prerequisites

1. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.
2. An alert rule is created. For more information, see Creating panels and alert in PCP Redis data source.

3. Configure SMTP and add a valid sender’s email address in the grafana/grafana.ini file:

   # vi /etc/grafana/grafana.ini
   
   [smtp]
   enabled = true
   from_address = abc@gmail.com

   Replace abc@gmail.com by a valid email address.

4. Restart grafana-server

   # systemctl restart grafana-server.service

Procedure

1. From the menu, hover over the       Alerting icon → click Notification channels → Add channel.

2. In the Add notification channel details pane, perform the following:

   1. Enter your name in the Name text box
   2. Select the communication Type, for example, Email and enter the email address. You can add multiple email addresses using the ; separator.
   3. Optional: Configure Optional Email settings and Notification settings.
3. Click Save.

4. Select a notification channel in the alert rule:

   1. From the menu, hover over the       Alerting icon and then click Alert rules.
   2. From the Alert Rules tab, click the created alert rule.
   3. On the Notifications tab, select your notification channel name from the Send to option, and then add an alert message.
   4. Click Apply.

Procedure

1. Install the sasl framework for the scram-sha-256 authentication mechanism:

   # yum install cyrus-sasl-scram cyrus-sasl-lib

2. Specify the supported authentication mechanism and the user database path in the pmcd.conf file:

   # vi /etc/sasl2/pmcd.conf
   
   mech_list: scram-sha-256
   
   sasldb_path: /etc/pcp/passwd.db

3. Create a new user:

   # useradd -r metrics

   Replace metrics by your user name.

4. Add the created user in the user database:

   # saslpasswd2 -a pmcd metrics
   
   Password:
   Again (for verification):

   To add the created user, you are required to enter the metrics account password.

5. Set the permissions of the user database:

   # chown root:pcp /etc/pcp/passwd.db
   # chmod 640 /etc/pcp/passwd.db

6. Restart the pmcd service:

   # systemctl restart pmcd

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.
3. The scram-sha-256 authentication mechanism is configured. For more information, see Setting up authentication between PCP components.

Procedure

1. Install the pcp-pmda-bpftrace package:

   # yum install pcp-pmda-bpftrace

2. Edit the bpftrace.conf file and add the user that you have created in Setting up authentication between PCP components:

   # vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf
   
   [dynamic_scripts]
   enabled = true
   auth_enabled = true
   allowed_users = root,metrics

   Replace metrics by your user name.

3. Install bpftrace PMDA:

   # cd /var/lib/pcp/pmdas/bpftrace/
   # ./Install
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check bpftrace metrics have appeared ... 7 metrics and 6 values

   The pmda-bpftrace is now installed, and can only be used after authenticating your user. For more information, see Viewing the PCP bpftrace System Analysis dashboard.

Prerequisites

1. The PCP bpftrace is installed. For more information, see Installing PCP bpftrace.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type bpftrace in the Filter by name or type text box and then click PCP bpftrace.

4. In the Data Sources / PCP bpftrace pane, perform the following:

   1. Add http://localhost:44322 in the URL field.
   2. Toggle the Basic Auth option and add the created user credentials in the User and Password field.

   3. Click Save & Test.

      Figure 10.5. Adding PCP bpftrace in the data source

      

   4. Click Dashboards tab → Import → PCP bpftrace: System Analysis to see a dashboard with an overview of any useful metrics.

      Figure 10.6. PCP bpftrace: System Analysis

      

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

1. Install the pcp-pmda-bcc package:

   # yum install pcp-pmda-bcc

2. Install the bcc PMDA:

   # cd /var/lib/pcp/pmdas/bcc
   # ./Install
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state.
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: Enabled modules:
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork',
   [...]
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values

Prerequisites

1. The PCP Vector is installed. For more information, see Installing PCP Vector.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type vector in the Filter by name or type text box and then click PCP Vector.

4. In the Data Sources / PCP Vector pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab → Import → PCP Vector: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 10.7. PCP Vector: Host Overview

      

5. From the menu, hover over the       Performance Co-Pilot plugin and then click PCP Vector Checklist.

   In the PCP checklist, click       help or       warning icon to view the related troubleshooting or reference links.

   Figure 10.8. Performance Co-Pilot / PCP Vector Checklist

   

Procedure

1. Hover the cursor over the Dashboards tab and click + New dashboard.
2. In the Add panel menu, click Add a new panel.

3. In the Query tab:

   1. Select PCP Redis from the query list instead of the selected default option.
   2. In the text field of A, enter a metric, for example, kernel.all.load to visualize the kernel load graph.
4. Click the visualization dropdown menu, which is set to Time series by default, and then click Heatmap.
5. Optional: In the Panel Options dropdown menu, add a Panel Title and Description.

6. In the Heatmap dropdown menu, under the Calculate from data setting, click Yes.

   Heatmap

   

7. Optional: In the Colors dropdown menu, change the Scheme from the default Orange and select the number of steps (color shades).

8. Optional: In the Tooltip dropdown menu, under the Show histogram (Y Axis) setting, click the toggle to display a cell’s position within its specific histogram when hovering your cursor over a cell in the heatmap. For example:

   Show histogram (Y Axis) cell display

   

Procedure

1. Log in to the RHEL 8 web console.

   For details, see Logging in to the web console.

2. Click Overview.

3. In the Configuration section, click the current performance profile.

   

4. In the Change Performance Profile dialog box, set the required profile.

   

5. Click Change Profile.

Procedure

1. Log in to the RHEL 8 web console.

   For details, see Logging in to the web console.

2. Click Overview.

3. In the Usage section, click View metrics and history.

   

   The Metrics and history section opens:

   • The current system configuration and usage:
   • The performance metrics in a graphical form over a user-specified time interval:

Procedure

1. Log in to the RHEL 8 web console.

   For details, see Logging in to the web console.

2. In the Overview page, click View metrics and history in the Usage table.
3. Click the Metrics settings button.

4. Move the Export to network slider to active position.

   

   If you do not have the redis package installed, the web console prompts you to install it.

5. To open the pmproxy service, select a zone from a drop-down list and click the Add pmproxy button.
6. Click Save.

Verification

1. Click Networking.
2. In the Firewall table, click the Edit rules and zones button.
3. Search for pmproxy in your selected zone.

Procedure

1. Optional: Select an existing TuneD profile on which your profile will be based. For a list of available profiles, see TuneD profiles distributed with RHEL.

   To see which profile is currently active, use:

   $ tuned-adm active

2. Create a new directory to hold your TuneD profile:

   # mkdir /etc/tuned/my-profile

3. Find the system unique identifier of the selected block device:

   $ udevadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)'
   
   ID_WWN=0x5002538d00000000_
   ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   ID_SERIAL_SHORT=20120501030900000

   Note

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

4. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following options:

   1. Optional: Include an existing profile:

      [main]
      include=existing-profile

   2. Set the selected disk scheduler for the device that matches the WWN identifier:

      [disk]
      devices_udev_regex=IDNAME=device system unique id
      elevator=selected-scheduler

      Here:

      • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).

      • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

        To match multiple devices in the devices_udev_regex option, enclose the identifiers in parentheses and separate them with vertical bars:

        devices_udev_regex=(ID_WWN=0x5002538d00000000)|(ID_WWN=0x1234567800000000)

5. Enable your profile:

   # tuned-adm profile my-profile

Verification

1. Verify that the TuneD profile is active and applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See TuneD log file ('/var/log/tuned/tuned.log') for details.

2. Read the contents of the /sys/block/device/queue/scheduler file:

   # cat /sys/block/device/queue/scheduler
   
   [mq-deadline] kyber bfq none

   In the file name, replace device with the block device name, for example sdc.

   The active scheduler is listed in square brackets ([]).

Procedure

1. Find the system unique identifier of the block device:

   $ udevadm info --name=/dev/device | grep -E '(WWN|SERIAL)'
   E: ID_WWN=0x5002538d00000000
   E: ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   E: ID_SERIAL_SHORT=20120501030900000

   Note

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

2. Configure the udev rule. Create the /etc/udev/rules.d/99-scheduler.rules file with the following content:

   ACTION=="add|change", SUBSYSTEM=="block", ENV{IDNAME}=="device system unique id", ATTR{queue/scheduler}="selected-scheduler"

   Here:

   • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).
   • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

3. Reload udev rules:

   # udevadm control --reload-rules

4. Apply the scheduler configuration:

   # udevadm trigger --type=devices --action=change

Procedure

1. Remove the server max protocol parameter from the [global] section in the /etc/samba/smb.conf file.

2. Reload the Samba configuration

   # smbcontrol all reload-config

Procedure

1. Rename all files on the share to lowercase.

   Note

   Using the settings in this procedure, files with names other than in lowercase will no longer be displayed.

2. Set the following parameters in the share’s section:

   case sensitive = true
   default case = lower
   preserve case = no
   short preserve case = no

   For details about the parameters, see their descriptions in the smb.conf(5) man page on your system.

3. Verify the /etc/samba/smb.conf file:

   # testparm

4. Reload the Samba configuration:

   # smbcontrol all reload-config

1. List the available TuneD profiles.

   # tuned-adm list
   
   Available profiles:
   - balanced             - General non-specialized TuneD profile
   - desktop              - Optimize for the desktop use-case
   [...]
   - virtual-guest        - Optimize for running inside a virtual guest
   - virtual-host         - Optimize for running KVM guests
   Current active profile: balanced

2. Optional: Create a new TuneD profile or edit an existing TuneD profile.

   For more information, see Customizing TuneD profiles.

3. Activate a TuneD profile.

   # tuned-adm profile selected-profile

   • To optimize a virtualization host, use the virtual-host profile.

     # tuned-adm profile virtual-host

   • On a RHEL guest operating system, use the virtual-guest profile.

     # tuned-adm profile virtual-guest

Verification

1. Display the active profile for TuneD.

   # tuned-adm active
   Current active profile: virtual-host

2. Ensure that the TuneD profile settings have been applied on your system.

   # tuned-adm verify
   Verification succeeded, current system settings match the preset profile. See tuned log file ('/var/log/tuned/tuned.log') for details.

1. Adjust how many host CPUs are assigned to the VM. You can do this using the CLI or the web console.

2. Ensure that the vCPU model is aligned with the CPU model of the host. For example, to set the testguest1 VM to use the CPU model of the host:

   # virt-xml testguest1 --edit --cpu host-model

3. Deactivate kernel same-page merging (KSM).

4. If your host machine uses Non-Uniform Memory Access (NUMA), you can also configure NUMA for its VMs. This maps the host’s CPU and memory processes onto the CPU and memory processes of the VM as closely as possible. In effect, NUMA tuning provides the vCPU with a more streamlined access to the system memory allocated to the VM, which can improve the vCPU processing effectiveness.

   For details, see Configuring NUMA in a virtual machine and Virtual machine performance optimization for specific workloads.