Monitoring and managing system status and performance

Procedure

1. Edit /etc/tuned/cpu-partitioning-variables.conf file and add the following information:

   # All isolated CPUs:
   isolated_cores=2-23
   # Isolated CPUs without the kernel’s scheduler load balancing:
   no_balance_cores=2,3

2. Set the cpu-partitioning TuneD profile:

   # tuned-adm profile cpu-partitioning

3. Reboot

   After rebooting, the system is tuned for low-latency, according to the isolation in the cpu-partitioning figure. The application can use taskset to pin the reader and writer threads to CPUs 2 and 3, and the remaining application threads on CPUs 4-23.

Procedure

1. Create the /etc/tuned/my_profile directory:

   # mkdir /etc/tuned/my_profile

2. Create a tuned.conf file in this directory, and add the following content:

   # vi /etc/tuned/my_profile/tuned.conf
   [main]
   summary=Customized tuning on top of cpu-partitioning
   include=cpu-partitioning
   [cpu]
   force_latency=cstate.id:0|1

3. Use the new profile:

   # tuned-adm profile my_profile

Procedure

1. Install the TuneD package:

   # dnf install tuned

2. Enable and start the TuneD service:

   # systemctl enable --now tuned

3. Optionally, install TuneD profiles for real-time systems:

   For the TuneD profiles for real-time systems enable rhel-9 repository.

   # subscription-manager repos --enable=rhel-9-for-x86_64-nfv-beta-rpms

   Install it.

   # dnf install tuned-profiles-realtime tuned-profiles-nfv

4. Verify that a TuneD profile is active and applied:

   $ tuned-adm active
   
   Current active profile: throughput-performance

   Note

   The active profile TuneD automatically presets differs based on your machine type and system settings.

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

Procedure

1. Optionally, you can let TuneD recommend the most suitable profile for your system:

   # tuned-adm recommend
   
   throughput-performance

2. Activate a profile:

   # tuned-adm profile selected-profile

   Alternatively, you can activate a combination of multiple profiles:

   # tuned-adm profile selected-profile1 selected-profile2

   Example 1.3. A virtual machine optimized for low power consumption

   The following example optimizes the system to run in a virtual machine with the best performance and concurrently tunes it for low power consumption, while the low power consumption is the priority:

   # tuned-adm profile virtual-guest powersave

3. View the current active TuneD profile on your system:

   # tuned-adm active
   
   Current active profile: selected-profile

4. Reboot the system:

   # reboot

Procedure

1. In the /etc/tuned/ directory, create a new directory named the same as the profile that you want to create:

   # mkdir /etc/tuned/my-profile

2. In the new directory, create a file named tuned.conf. Add a [main] section and plug-in definitions in it, according to your requirements.

   For example, see the configuration of the balanced profile:

   [main]
   summary=General non-specialized TuneD profile
   
   [cpu]
   governor=conservative
   energy_perf_bias=normal
   
   [audio]
   timeout=10
   
   [video]
   radeon_powersave=dpm-balanced, auto
   
   [scsi_host]
   alpm=medium_power

3. To activate the profile, use:

   # tuned-adm profile my-profile

4. Verify that the TuneD profile is active and the system settings are applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

Procedure

1. In the /etc/tuned/ directory, create a new directory named the same as the profile that you want to create:

   # mkdir /etc/tuned/modified-profile

2. In the new directory, create a file named tuned.conf, and set the [main] section as follows:

   [main]
   include=parent-profile

   Replace parent-profile with the name of the profile you are modifying.

3. Include your profile modifications.

   Example 2.18. Lowering swappiness in the throughput-performance profile

   To use the settings from the throughput-performance profile and change the value of vm.swappiness to 5, instead of the default 10, use:

   [main]
   include=throughput-performance
   
   [sysctl]
   vm.swappiness=5

4. To activate the profile, use:

   # tuned-adm profile modified-profile

5. Verify that the TuneD profile is active and the system settings are applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See tuned log file ('/var/log/tuned/tuned.log') for details.

Procedure

1. Optional: Select an existing TuneD profile on which your profile will be based. For a list of available profiles, see TuneD profiles distributed with RHEL.

   To see which profile is currently active, use:

   $ tuned-adm active

2. Create a new directory to hold your TuneD profile:

   # mkdir /etc/tuned/my-profile

3. Find the system unique identifier of the selected block device:

   $ udevadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)'
   
   ID_WWN=0x5002538d00000000_
   ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   ID_SERIAL_SHORT=20120501030900000

   Note

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

4. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following options:

   1. Optional: Include an existing profile:

      [main]
      include=existing-profile

   2. Set the selected disk scheduler for the device that matches the WWN identifier:

      [disk]
      devices_udev_regex=IDNAME=device system unique id
      elevator=selected-scheduler

      Here:

      • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).

      • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

        To match multiple devices in the devices_udev_regex option, enclose the identifiers in parentheses and separate them with vertical bars:

        devices_udev_regex=(ID_WWN=0x5002538d00000000)|(ID_WWN=0x1234567800000000)

5. Enable your profile:

   # tuned-adm profile my-profile

Verification

1. Verify that the TuneD profile is active and applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See TuneD log file ('/var/log/tuned/tuned.log') for details.

2. Read the contents of the /sys/block/device/queue/scheduler file:

   # cat /sys/block/device/queue/scheduler
   
   [mq-deadline] kyber bfq none

   In the file name, replace device with the block device name, for example sdc.

   The active scheduler is listed in square brackets ([]).

Procedure

1. View the current policies and priorities:

   # tuna show_threads
   pid   SCHED_ rtpri affinity             cmd
   1      OTHER     0      0,1            init
   2       FIFO    99        0     migration/0
   3      OTHER     0        0     ksoftirqd/0
   4       FIFO    99        0      watchdog/0

   Alternatively, to view a specific thread corresponding to a PID or matching a command name, enter:

   # tuna show_threads -t pid_or_cmd_list

   The pid_or_cmd_list argument is a list of comma-separated PIDs or command-name patterns.

2. Depending on you scenario, perform one of the following actions:

   • To tune CPUs by using the tuna CLI, complete the steps in Tuning CPUs by using the tuna tool.
   • To tune the IRQs by using the tuna tool, complete the steps in Tuning IRQs by using the tuna tool.

3. Save the changed configuration:

   # tuna save filename

   This command saves only currently running kernel threads. Processes that are not running are not saved.

Procedure

1. List all CPUs and specify the list of CPUs to be affected by the command:

   # ps ax | awk 'BEGIN { ORS="," }{ print $1 }'
   PID,1,2,3,4,5,6,8,10,11,12,13,14,15,16,17,19

2. Display the thread list in the tuna interface:

   # tuna show_threads -t 'thread_list from above cmd'

3. Specify the list of CPUs to be affected by a command:

   # *tuna [command] --cpus cpu_list *

   The cpu_list argument is a list of comma-separated CPU numbers, for example, --cpus 0,2.

   To add a specific CPU to the current cpu_list, use, for example, --cpus +0.

4. Depending on your scenario, perform one of the following actions:

   • To isolate a CPU, enter:

     # tuna isolate --cpus cpu_list

   • To include a CPU, enter:

     # tuna include --cpus cpu_list

5. To use a system with four or more processors, make all ssh threads run on CPU 0 and 1 and all http threads on CPU 2 and 3:

   # tuna move --cpus 0,1 -t ssh*
   # tuna move --cpus 2,3 -t http\*

Procedure

1. View the current IRQs and their affinity:

   # tuna show_irqs
   # users            affinity
   0 timer                   0
   1 i8042                   0
   7 parport0                0

2. Specify the list of IRQs to be affected by a command:

   # tuna [command] --irqs irq_list --cpus cpu_list

   The irq_list argument is a list of comma-separated IRQ numbers or user-name patterns.

   Replace [command] with, for example, --spread.

3. Move an interrupt to a specified CPU:

   # tuna show_irqs --irqs 128
   users            affinity
   128 iwlwifi           0,1,2,3
   
   # tuna move --irqs 128 --cpus 3

   Replace 128 with the irq_list argument and 3 with the cpu_list argument.

   The cpu_list argument is a list of comma-separated CPU numbers, for example, --cpus 0,2. For more information, see Tuning CPUs by using the tuna tool.

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Manage metrics
     hosts: localhost
     roles:
       - rhel-system-roles.metrics
     vars:
       metrics_graph_service: yes
       metrics_manage_firewall: true
       metrics_manage_selinux: true

   Because the metrics_graph_service boolean is set to value="yes", Grafana is automatically installed and provisioned with pcp added as a data source. Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux system roles to manage the ports used by the metrics role.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Configure a fleet of machines to monitor themselves
     hosts: managed-node-01.example.com
     roles:
       - rhel-system-roles.metrics
     vars:
       metrics_retention_days: 0
       metrics_manage_firewall: true
       metrics_manage_selinux: true

   Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux roles to manage the ports used by the metrics role.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   - name: Set up your local machine to centrally monitor a fleet of machines
     hosts: localhost
     roles:
       - rhel-system-roles.metrics
     vars:
       metrics_graph_service: yes
       metrics_query_service: yes
       metrics_retention_days: 10
       metrics_monitored_hosts: ["database.example.com", "webserver.example.com"]
       metrics_manage_firewall: yes
       metrics_manage_selinux: yes

   Because the metrics_graph_service and metrics_query_service booleans are set to value="yes", grafana is automatically installed and provisioned with pcp added as a data source with the pcp data recording indexed into redis, allowing the pcp querying language to be used for complex querying of the data. Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux roles to manage the ports used by the metrics role.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Procedure

1. Edit an existing playbook file, for example ~/playbook.yml, and add the authentication-related variables:

   ---
   - name: Set up authentication by using the scram-sha-256 authentication mechanism
     hosts: managed-node-01.example.com
     roles:
       - rhel-system-roles.metrics
     vars:
       metrics_retention_days: 0
       metrics_manage_firewall: true
       metrics_manage_selinux: true
       metrics_username: <username>
       metrics_password: <password>

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Configure and enable metrics collection for Microsoft SQL Server
     hosts: localhost
     roles:
       - rhel-system-roles.metrics
     vars:
       metrics_from_mssql: true
       metrics_manage_firewall: true
       metrics_manage_selinux: true

   Because metrics_manage_firewall and metrics_manage_selinux are both set to true, the metrics role uses the firewall and selinux roles to manage the ports used by the metrics role.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Procedure

1. Create a playbook file, for example ~/playbook.yml, with the following content:

   ---
   - name: Configure PCP webhooks
     hosts: servers
     tasks:
       - name: Configure PCP metrics recording
         ansible.builtin.include_role:
           name: rhel-system-roles.metrics
         vars:
           metrics_retention_days: 7
           metrics_manage_firewall: true
   
   
   - name: Configure the PMIE webhooks
     hosts: metrics_monitor
     tasks:
       - name: Configure the monitoring node
         ansible.builtin.include_role:
           name: redhat.rhel_system_roles.metrics
         vars:
           metrics_manage_firewall: true
           metrics_retention_days: 7
           metrics_monitored_hosts:  "{{ groups['servers'] }}"
           metrics_webhook_endpoint: "http://<webserver>:<port>/<endpoint>"

   The settings specified in the example playbook include the following:

   metrics_manage_firewall

   When true, the firewall RHEL system role manages the ports used by the metrics role.

   metrics_retention_days

   Number of days to keep the collected metrics.

   metrics_monitored_hosts

   Hosts that the monitoring system should observe.

   metrics_webhook_endpoint

   A webhook endpoint where notifications of any detected performance issues are sent. By default, these detections are logged to the local system only.

   For details about all variables used in the playbook, see the /usr/share/ansible/roles/rhel-system-roles.metrics/README.md file on the control node. The playbook configures the pcp-monitor-node-01.example.com host as the central monitoring site for itself and the server-node-01.example.com and server-node-02.example.com systems. The playbook also configures the global webhook_action and global webhook_endpoint PMIE configuration options for all 3 systems and restarts the PMIE service to apply the changes.

2. Validate the playbook syntax:

   $ ansible-playbook --syntax-check ~/playbook.yml

   Note that this command only validates the syntax and does not protect against a wrong but valid configuration.

3. Run the playbook:

   $ ansible-playbook ~/playbook.yml

Verification

1. Check the configuration summary on pcp-monitor-node-01.example.com:

   [root@pcp-monitor-node-01 ~]# pcp summary
   Performance Co-Pilot configuration on pcp-monitor-node-01.example.com:
   
    platform: Linux pcp-monitor-node-01.example.com 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 01:51:18 EST 2024 x86_64
    hardware: 8 cpus, 1 disk, 1 node, 1773MB RAM
    timezone: CEST-2
    services: pmcd pmproxy
    pmcd: Version 6.2.0-1, 12 agents, 6 clients
    pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm jbd2
          	dm openmetrics
    pmlogger: primary logger: /var/log/pcp/pmlogger/pcp-monitor-node-01.example.com/20240510.16.25
              server-node-01.example.com: /var/log/pmlogger/server-node-01.example.com/20240510.16.25
              server-node-02.example.com: /var/log/pmlogger/server-node-02.example.com/20240510.16.25
    pmie: primary engine: /var/log/pcp/pmie/pcp-monitor-node-01.example.com/pmie.log
          server-node-01.example.com: : /var/log/pcp/pmie/server-node-01.example.com/pmie.log
          server-node-02.example.com: : /var/log/pcp/pmie/server-node-02.example.com/pmie.log

   The final three lines of the summary show that PMIE is configured to monitor all three systems.

2. Verify that the global webhook_action PMIE configuration option is enabled:

   [root@pcp-monitor-node-01 ~]# grep webbook_action /var/lib/pcp/config/pmie/config.default
   // 0 global webhook_action = yes

Procedure

1. Install the pcp package:

   # dnf install pcp

2. Enable and start the pmcd service on the host machine:

   # systemctl enable pmcd
   
   # systemctl start pmcd

Procedure

1. Update the pmlogger configuration:

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

2. Start the pmcd and pmlogger services:

   # systemctl start pmcd.service
   
   # systemctl start pmlogger.service

3. Execute the required operations to record the performance data.

4. Stop the pmcd and pmlogger services:

   # systemctl stop pmcd.service
   
   # systemctl stop pmlogger.service

5. Save the output and save it to a tar.gz file named based on the host name and the current date and time:

   # cd /var/log/pcp/pmlogger/
   
   # tar -czf $(hostname).$(date +%F-%Hh%M).pcp.tar.gz $(hostname)

   Extract this file and analyze the data using PCP tools.

Prerequisites

1. Install the pcp-pmda-redis package:

   # dnf install pcp-pmda-redis

2. Install the redis PMDA:

   # cd /var/lib/pcp/pmdas/redis && ./Install

Procedure

1. Create or modify the pmlogger configuration file:

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

2. Follow pmlogconf prompts to enable or disable groups of related performance metrics and to control the logging interval for each enabled group.

Procedure

1. Install the pcp-system-tools package:

   # dnf install pcp-system-tools

2. Configure an IP address for pmcd:

   # echo "-i 192.168.4.62" >>/etc/pcp/pmcd/pmcd.options

   Replace 192.168.4.62 with the IP address, the client should listen on.

   By default, pmcd is listening on the localhost.

3. Configure the firewall to add the public zone permanently:

   # firewall-cmd --permanent --zone=public --add-port=44321/tcp
   success
   
   # firewall-cmd --reload
   success

4. Set an SELinux boolean:

   # setsebool -P pcp_bind_all_unreserved_ports on

5. Enable the pmcd and pmlogger services:

   # systemctl enable pmcd pmlogger
   # systemctl restart pmcd pmlogger

Procedure

1. Install the pcp-system-tools package:

   # dnf install pcp-system-tools

2. Create the /etc/pcp/pmlogger/control.d/remote file with the following content:

   # DO NOT REMOVE OR EDIT THE FOLLOWING LINE
   $version=1.1
   
   192.168.4.13 n n PCP_ARCHIVE_DIR/rhel7u4a -r -T24h10m -c config.rhel7u4a
   192.168.4.14 n n PCP_ARCHIVE_DIR/rhel6u10a -r -T24h10m -c config.rhel6u10a
   192.168.4.62 n n PCP_ARCHIVE_DIR/rhel8u1a -r -T24h10m -c config.rhel8u1a
   192.168.4.69 n n PCP_ARCHIVE_DIR/rhel9u3a -r -T24h10m -c config.rhel9u3a

   Replace 192.168.4.13, 192.168.4.14, 192.168.4.62 and 192.168.4.69 with the client IP addresses.

3. Enable the pmcd and pmlogger services:

   # systemctl enable pmcd pmlogger
   # systemctl restart pmcd pmlogger

Procedure

1. Open the /etc/pcp.conf file in a text editor of your choice and set the PCP archive version:

   PCP_ARCHIVE_VERSION=3

2. Restart the pmlogger service to apply your configuration changes:

   # systemctl restart pmlogger.service

3. Create a new PCP archive log using your new configuration. For more information, see Logging performance data with pmlogger.

Procedure

1. Install the following packages:

   1. Install the pcp-system-tools:

      # dnf install pcp-system-tools

   2. Install the pmda-postfix package to monitor postfix:

      # dnf install pcp-pmda-postfix postfix

   3. Install the logging daemon:

      # dnf install rsyslog

   4. Install the mail client for testing:

      # dnf install mutt

2. Enable the postfix and rsyslog services:

   # systemctl enable postfix rsyslog
   # systemctl restart postfix rsyslog

3. Enable the SELinux boolean, so that pmda-postfix can access the required log files:

   # setsebool -P pcp_read_generic_logs=on

4. Install the PMDA:

   # cd /var/lib/pcp/pmdas/postfix/
   
   # ./Install
   
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Waiting for pmcd to terminate ...
   Starting pmcd ...
   Check postfix metrics have appeared ... 7 metrics and 58 values

Procedure

1. Launch the PCP Charts application from the command line:

   # pmchart

   Figure 7.1. PCP Charts application

   

   The pmtime server settings are located at the bottom. The start and pause button allows you to control:

   • The interval in which PCP polls the metric data
   • The date and time for the metrics of historical data
2. Click File and then New Chart to select metric from both the local machine and remote machines by specifying their host name or address. Advanced configuration options include the ability to manually set the axis values for the chart, and to manually choose the color of the plots.

3. Record the views created in the PCP Charts application:

   Following are the options to take images or record the views created in the PCP Charts application:

   • Click File and then Export to save an image of the current view.
   • Click Record and then Start to start a recording. Click Record and then Stop to stop the recording. After stopping the recording, the recorded metrics are archived to be viewed later.

4. Optional: In the PCP Charts application, the main configuration file, known as the view, allows the metadata associated with one or more charts to be saved. This metadata describes all chart aspects, including the metrics used and the chart columns. Save the custom view configuration by clicking File and then Save View, and load the view configuration later.

   The following example of the PCP Charts application view configuration file describes a stacking chart graph showing the total number of bytes read and written to the given XFS file system loop1:

   #kmchart
   version 1
   
   chart title "Filesystem Throughput /loop1" style stacking antialiasing off
       plot legend "Read rate"   metric xfs.read_bytes   instance  "loop1"
       plot legend "Write rate"  metric xfs.write_bytes  instance  "loop1"

Procedure

1. Install PCP:

   # dnf install pcp-zeroconf

2. Install packages required for the pyodbc driver:

   # dnf install python3-pyodbc

3. Install the mssql agent:

   1. Install the Microsoft SQL Server domain agent for PCP:

      # dnf install pcp-pmda-mssql

   2. Edit the /etc/pcp/mssql/mssql.conf file to configure the SQL server account’s username and password for the mssql agent. Ensure that the account you configure has access rights to performance data.

      username: user_name
      password: user_password

      Replace user_name with the SQL Server account and user_password with the SQL Server user password for this account.

4. Install the agent:

   # cd /var/lib/pcp/pmdas/mssql
   # ./Install
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check mssql metrics have appeared ... 168 metrics and 598 values
   [...]

Procedure

1. Navigate to the xfs directory:

   # cd /var/lib/pcp/pmdas/xfs/

2. Install the XFS PMDA manually:

   xfs]# ./Install
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check xfs metrics have appeared ... 387 metrics and 387 values

Procedure

1. Display the value of a metric:

   $ pminfo -f xfs.write
   
   xfs.write
       value 325262

2. Reset all the XFS metrics:

   # pmstore xfs.control.reset 1
   
   xfs.control.reset old value=0 new value=1

Procedure

1. Install the following packages:

   # dnf install grafana grafana-pcp

2. Restart and enable the following service:

   # systemctl restart grafana-server
   # systemctl enable grafana-server

3. Open the server’s firewall for network traffic to the Grafana service.

   # firewall-cmd --permanent --add-service=grafana
   success
   
   # firewall-cmd --reload
   success

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

1. On the client system, open a browser and access the grafana-server on port 3000, using http://192.0.2.0:3000 link.

   Replace 192.0.2.0 with your machine IP.

2. For the first login, enter admin in both the Email or username and Password field.

   Grafana prompts to set a New password to create a secured account. If you want to set it later, click Skip.

3. From the menu, hover over the       Configuration icon and then click Plugins.
4. In the Plugins tab, type performance co-pilot in the Search by name or type text box and then click Performance Co-Pilot (PCP) plugin.
5. In the Plugins / Performance Co-Pilot pane, click Enable.

6. Click Grafana       icon. The Grafana Home page is displayed.

   Figure 9.1. Home Dashboard

   
   Note

   The top corner of the screen has a similar       icon, but it controls the general Dashboard settings.

7. In the Grafana Home page, click Add your first data source to add PCP Redis, PCP bpftrace, and PCP Vector data sources. For more information about adding data source, see:

   • To add pcp redis data source, view default dashboard, create a panel, and an alert rule, see Creating panels and alert in PCP Redis data source.
   • To add pcp bpftrace data source and view the default dashboard, see Viewing the PCP bpftrace System Analysis dashboard.
   • To add pcp vector data source, view the default dashboard, and to view the vector checklist, see Viewing the PCP Vector Checklist.
8. Optional: From the menu, hover over the admin profile       icon to change the Preferences including Edit Profile, Change Password, or to Sign out.

Procedure

1. As a root user, open the /etc/grafana/grana.ini file and adjust the following options in the [server] section to reflect the following:

   protocol = https
   cert_key = /etc/grafana/grafana.key
   cert_file = /etc/grafana/grafana.crt

2. Ensure grafana can access the certificates:

   # su grafana -s /bin/bash -c \
     'ls -1 /etc/grafana/grafana.crt /etc/grafana/grafana.key'
   /etc/grafana/grafana.crt
   /etc/grafana/grafana.key

3. Restart and enable the Grafana service to apply the configuration changes:

   # systemctl restart grafana-server
   # systemctl enable grafana-server

Verification

1. On the client system, open a browser and access the grafana-server machine on port 3000, using the https://192.0.2.0:3000 link. Replace 192.0.2.0 with your machine IP.

2. Confirm the       lock icon is displayed beside the address bar.

   Note

   If the protocol is set to http and an HTTPS connection is attempted, you will receive a ERR_SSL_PROTOCOL_ERROR error. If the protocol is set to https and an HTTP connection is attempted, the Grafana server responds with a “Client sent an HTTP request to an HTTPS server” message.

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.
3. Mail transfer agent, for example, sendmail or postfix is installed and configured.

Procedure

1. Install the redis package:

   # dnf install redis

2. Start and enable the following services:

   # systemctl start pmproxy redis
   # systemctl enable pmproxy redis

3. Restart the grafana-server:

   # systemctl restart grafana-server

Procedure

1. As a root user, open the /etc/redis/redis.conf file and adjust the TLS/SSL options to reflect the following properties:

   port 0
   tls-port 6379
   tls-cert-file /etc/redis/redis.crt
   tls-key-file /etc/redis/redis.key
   tls-client-key-file /etc/redis/client.key
   tls-client-cert-file /etc/redis/client.crt
   tls-ca-cert-file /etc/redis/ca.crt

2. Ensure redis can access the TLS certificates:

   # su redis -s /bin/bash -c \
     'ls -1 /etc/redis/ca.crt /etc/redis/redis.key /etc/redis/redis.crt'
   /etc/redis/ca.crt
   /etc/redis/redis.crt
   /etc/redis/redis.key

3. Restart the redis server to apply the configuration changes:

   # systemctl restart redis

Prerequisites

1. The PCP Redis is configured. For more information, see Configuring PCP Redis.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type redis in the Filter by name or type text box and then click PCP Redis.

4. In the Data Sources / PCP Redis pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab → Import → PCP Redis: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 9.2. PCP Redis: Host Overview

      

5. Add a new panel:

   1. From the menu, hover over the       Create icon → Dashboard → Add new panel icon to add a panel.
   2. In the Query tab, select the PCP Redis from the query list instead of the selected default option and in the text field of A, enter metric, for example, kernel.all.load to visualize the kernel load graph.
   3. Optional: Add Panel title and Description, and update other options from the Settings.
   4. Click Save to apply changes and save the dashboard. Add Dashboard name.

   5. Click Apply to apply changes and go back to the dashboard.

      Figure 9.3. PCP Redis query panel

      

6. Create an alert rule:

   1. In the PCP Redis query panel, click       Alert and then click Create Alert.
   2. Edit the Name, Evaluate query, and For fields from the Rule, and specify the Conditions for your alert.

   3. Click Save to apply changes and save the dashboard. Click Apply to apply changes and go back to the dashboard.

      Figure 9.4. Creating alerts in the PCP Redis panel

      
   4. Optional: In the same panel, scroll down and click Delete icon to delete the created rule.

   5. Optional: From the menu, click       Alerting icon to view the created alert rules with different alert statuses, to edit the alert rule, or to pause the existing rule from the Alert Rules tab.

      To add a notification channel for the created alert rule to receive an alert notification from Grafana, see Adding notification channels for alerts.

Prerequisites

1. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.
2. An alert rule is created. For more information, see Creating panels and alert in PCP Redis data source.

3. Configure SMTP and add a valid sender’s email address in the grafana/grafana.ini file:

   # vi /etc/grafana/grafana.ini
   
   [smtp]
   enabled = true
   from_address = abc@gmail.com

   Replace abc@gmail.com by a valid email address.

4. Restart grafana-server

   # systemctl restart grafana-server.service

Procedure

1. From the menu, hover over the       Alerting icon → click Contact Points → New contact point.

2. In the New contact point details view, perform the following:

   1. Enter your name in the Name text box
   2. Select the Contact point type, for example, Email and enter the email address. You can add many email addresses by using the ; separator.
   3. Optional: Configure Optional Email settings and Notification settings.
3. Click Save contact point.

4. Select a notification channel in the alert rule:

   1. From the menu, select Notification policies icon and then click + New specific policy.
   2. Choose the Contact point you have just created
   3. Click the Save policy button

Procedure

1. Install the sasl framework for the scram-sha-256 authentication mechanism:

   # dnf install cyrus-sasl-scram cyrus-sasl-lib

2. Specify the supported authentication mechanism and the user database path in the pmcd.conf file:

   # vi /etc/sasl2/pmcd.conf
   
   mech_list: scram-sha-256
   
   sasldb_path: /etc/pcp/passwd.db

3. Create a new user:

   # useradd -r metrics

   Replace metrics by your user name.

4. Add the created user in the user database:

   # saslpasswd2 -a pmcd metrics
   
   Password:
   Again (for verification):

   To add the created user, you are required to enter the metrics account password.

5. Set the permissions of the user database:

   # chown root:pcp /etc/pcp/passwd.db
   # chmod 640 /etc/pcp/passwd.db

6. Restart the pmcd service:

   # systemctl restart pmcd

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.
3. The scram-sha-256 authentication mechanism is configured. For more information, see Setting up authentication between PCP components.

Procedure

1. Install the pcp-pmda-bpftrace package:

   # dnf install pcp-pmda-bpftrace

2. Edit the bpftrace.conf file and add the user that you have created in the {setting-up-authentication-between-pcp-components}:

   # vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf
   
   [dynamic_scripts]
   enabled = true
   auth_enabled = true
   allowed_users = root,metrics

   Replace metrics by your user name.

3. Install bpftrace PMDA:

   # cd /var/lib/pcp/pmdas/bpftrace/
   # ./Install
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check bpftrace metrics have appeared ... 7 metrics and 6 values

   The pmda-bpftrace is now installed, and can only be used after authenticating your user. For more information, see Viewing the PCP bpftrace System Analysis dashboard.

Prerequisites

1. The PCP bpftrace is installed. For more information, see Installing PCP bpftrace.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type bpftrace in the Filter by name or type text box and then click PCP bpftrace.

4. In the Data Sources / PCP bpftrace pane, perform the following:

   1. Add http://localhost:44322 in the URL field.
   2. Toggle the Basic Auth option and add the created user credentials in the User and Password field.

   3. Click Save & Test.

      Figure 9.5. Adding PCP bpftrace in the data source

      

   4. Click Dashboards tab → Import → PCP bpftrace: System Analysis to see a dashboard with an overview of any useful metrics.

      Figure 9.6. PCP bpftrace: System Analysis

      

Prerequisites

1. PCP is configured. For more information, see Setting up PCP with pcp-zeroconf.
2. The grafana-server is configured. For more information, see Setting up a grafana-server.

Procedure

1. Install the pcp-pmda-bcc package:

   # dnf install pcp-pmda-bcc

2. Install the bcc PMDA:

   # cd /var/lib/pcp/pmdas/bcc
   # ./Install
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: Initializing, currently in 'notready' state.
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: Enabled modules:
   [Wed Apr  1 00:27:48] pmdabcc(22341) Info: ['biolatency', 'sysfork',
   [...]
   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check bcc metrics have appeared ... 1 warnings, 1 metrics and 0 values

Prerequisites

1. The PCP Vector is installed. For more information, see Installing PCP Vector.
2. The grafana-server is accessible. For more information, see Accessing the Grafana web UI.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type vector in the Filter by name or type text box and then click PCP Vector.

4. In the Data Sources / PCP Vector pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab → Import → PCP Vector: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 9.7. PCP Vector: Host Overview

      

5. From the menu, hover over the       Performance Co-Pilot plugin and then click PCP Vector Checklist.

   In the PCP checklist, click       help or       warning icon to view the related troubleshooting or reference links.

   Figure 9.8. Performance Co-Pilot / PCP Vector Checklist

   

Procedure

1. Hover the cursor over the Dashboards tab and click + New dashboard.
2. In the Add panel menu, click Add a new panel.

3. In the Query tab:

   1. Select PCP Redis from the query list instead of the selected default option.
   2. In the text field of A, enter a metric, for example, kernel.all.load to visualize the kernel load graph.
4. Click the visualization dropdown menu, which is set to Time series by default, and then click Heatmap.
5. Optional: In the Panel Options dropdown menu, add a Panel Title and Description.

6. In the Heatmap dropdown menu, under the Calculate from data setting, click Yes.

   Heatmap

   

7. Optional: In the Colors dropdown menu, change the Scheme from the default Orange and select the number of steps (color shades).

8. Optional: In the Tooltip dropdown menu, under the Show histogram (Y Axis) setting, click the toggle to display a cell’s position within its specific histogram when hovering your cursor over a cell in the heatmap. For example:

   Show histogram (Y Axis) cell display

   

Procedure

1. Log in to the RHEL 9 web console.

   For details, see Logging in to the web console.

2. Click Overview.

3. In the Configuration section, click the current performance profile.

   

4. In the Change Performance Profile dialog box, set the required profile.

   

5. Click Change Profile.

Procedure

1. Log in to the RHEL 9 web console.

   For details, see Logging in to the web console.

2. Click Overview.

3. In the Usage section, click View metrics and history.

   

   The Metrics and history section opens:

   • The current system configuration and usage:
   • The performance metrics in a graphical form over a user-specified time interval:

Procedure

1. Log in to the RHEL 9 web console.

   For details, see Logging in to the web console.

2. In the Overview page, click View metrics and history in the Usage table.
3. Click the Metrics settings button.

4. Move the Export to network slider to active position.

   

   If you do not have the redis package installed, the web console prompts you to install it.

5. To open the pmproxy service, select a zone from a drop-down list and click the Add pmproxy button.
6. Click Save.

Verification

1. Click Networking.
2. In the Firewall table, click the Edit rules and zones button.
3. Search for pmproxy in your selected zone.

Procedure

1. Optional: Select an existing TuneD profile on which your profile will be based. For a list of available profiles, see TuneD profiles distributed with RHEL.

   To see which profile is currently active, use:

   $ tuned-adm active

2. Create a new directory to hold your TuneD profile:

   # mkdir /etc/tuned/my-profile

3. Find the system unique identifier of the selected block device:

   $ udevadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)'
   
   ID_WWN=0x5002538d00000000_
   ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   ID_SERIAL_SHORT=20120501030900000

   Note

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

4. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following options:

   1. Optional: Include an existing profile:

      [main]
      include=existing-profile

   2. Set the selected disk scheduler for the device that matches the WWN identifier:

      [disk]
      devices_udev_regex=IDNAME=device system unique id
      elevator=selected-scheduler

      Here:

      • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).

      • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

        To match multiple devices in the devices_udev_regex option, enclose the identifiers in parentheses and separate them with vertical bars:

        devices_udev_regex=(ID_WWN=0x5002538d00000000)|(ID_WWN=0x1234567800000000)

5. Enable your profile:

   # tuned-adm profile my-profile

Verification

1. Verify that the TuneD profile is active and applied:

   $ tuned-adm active
   
   Current active profile: my-profile

   $ tuned-adm verify
   
   Verification succeeded, current system settings match the preset profile.
   See TuneD log file ('/var/log/tuned/tuned.log') for details.

2. Read the contents of the /sys/block/device/queue/scheduler file:

   # cat /sys/block/device/queue/scheduler
   
   [mq-deadline] kyber bfq none

   In the file name, replace device with the block device name, for example sdc.

   The active scheduler is listed in square brackets ([]).

Procedure

1. Find the system unique identifier of the block device:

   $ udevadm info --name=/dev/device | grep -E '(WWN|SERIAL)'
   E: ID_WWN=0x5002538d00000000
   E: ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   E: ID_SERIAL_SHORT=20120501030900000

   Note

   The command in the this example will return all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although it is preferred to use a WWN, the WWN is not always available for a given device and any values returned by the example command are acceptable to use as the device system unique ID.

2. Configure the udev rule. Create the /etc/udev/rules.d/99-scheduler.rules file with the following content:

   ACTION=="add|change", SUBSYSTEM=="block", ENV{IDNAME}=="device system unique id", ATTR{queue/scheduler}="selected-scheduler"

   Here:

   • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).
   • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

3. Reload udev rules:

   # udevadm control --reload-rules

4. Apply the scheduler configuration:

   # udevadm trigger --type=devices --action=change

Procedure

1. Remove the server max protocol parameter from the [global] section in the /etc/samba/smb.conf file.

2. Reload the Samba configuration

   # smbcontrol all reload-config

Procedure

1. Rename all files on the share to lowercase.

   Note

   Using the settings in this procedure, files with names other than in lowercase will no longer be displayed.

2. Set the following parameters in the share’s section:

   case sensitive = true
   default case = lower
   preserve case = no
   short preserve case = no

   For details about the parameters, see their descriptions in the smb.conf(5) man page.

3. Verify the /etc/samba/smb.conf file:

   # testparm

4. Reload the Samba configuration:

   # smbcontrol all reload-config

1. List the available TuneD profiles.

   # tuned-adm list
   
   Available profiles:
   - balanced             - General non-specialized TuneD profile
   - desktop              - Optimize for the desktop use-case
   [...]
   - virtual-guest        - Optimize for running inside a virtual guest
   - virtual-host         - Optimize for running KVM guests
   Current active profile: balanced

2. Optional: Create a new TuneD profile or edit an existing TuneD profile.

   For more information, see Customizing TuneD profiles.

3. Activate a TuneD profile.

   # tuned-adm profile selected-profile

   • To optimize a virtualization host, use the virtual-host profile.

     # tuned-adm profile virtual-host

   • On a RHEL guest operating system, use the virtual-guest profile.

     # tuned-adm profile virtual-guest