Monitoring and managing system status and performance

Procedure

1. Install the TuneD package:

   # dnf install tuned

2. Enable and start the TuneD service:

   # systemctl enable --now tuned

3. Optional: Install TuneD profiles for real-time systems:

   For the TuneD profiles for real-time systems enable the rhel-10 repository.

   # subscription-manager repos --enable=rhel-10-for-x86_64-nfv-beta-rpms

   # dnf install tuned-profiles-realtime tuned-profiles-nfv

Procedure

1. List all the available predefined TuneD profiles:

   $ tuned-adm list

2. Optional: To let TuneD recommend the most suitable profile for your system, use the following command:

   # tuned-adm recommend

   throughput-performance

3. Activate the profile:

   # tuned-adm profile selected-profile

   Alternatively, you can activate a combination of multiple profiles:

   # tuned-adm profile selected-profile1 selected-profile2

4. View the current active TuneD profile on your system:

   # tuned-adm active

   Current active profile: selected-profile

5. Disable all tunings temporarily:

   # tuned-adm off

   The tunings are applied again after the TuneD service restarts.

6. Optional: Stop and disable the TuneD service permanently:

   # systemctl disable --now tuned

Procedure

1. In the /etc/tuned/profiles directory, create a new directory named the same as the profile that you want to create:

   # mkdir /etc/tuned/profiles/my-profile

2. In the new directory, create a file named tuned.conf.

3. Edit the file and add a [main] section and plug in definitions in it, according to your requirements. For example, see the configuration of the balanced profile:

   [main]
   summary=General non-specialized TuneD profile
   
   [cpu]
   governor=conservative
   energy_perf_bias=normal
   
   [audio]
   timeout=10
   
   [video]
   radeon_powersave=dpm-balanced, auto
   
   [scsi_host]
   alpm=medium_power

4. Activate the profile.

   # tuned-adm profile my-profile

Procedure

1. In the /etc/tuned/profiles directory, create a new directory named the same as the profile that you want to create:

   # mkdir /etc/tuned/profiles/modified-profile

2. In the new directory, create a file named tuned.conf, and set the [main] section as follows:

   [main]
   include=parent-profile

   Replace parent-profile with the name of the profile you are modifying, for example: throughput-performance

3. Include your profile modifications. For example, to lower swappiness in the throughput-performance profile, change the value of vm.swappiness to 5, instead of the default 10, use:

   [main]
   include=throughput-performance
   
   [sysctl]
   vm.swappiness=5

4. Activate the profile.

   # tuned-adm profile modified-profile

Procedure

1. Edit the /etc/tuned/tuned-main.conf file with the following changes:

   sudo vi /etc/tuned/tuned-main.conf

   daemon = 0

2. Save and close the file.

3. Restart the tuned service or reapply or switch profile:

   # systemctl restart tuned

   # tuned-adm profile <tuned_profile>

4. Verify the tuned status:

   # tuned-adm active

   It seems that tuned daemon is not running, preset profile is not activated.
   Preset profile: <tuned_profile>

Procedure

1. Optional: Select an existing TuneD profile on which your profile will be based. For a list of available profiles, see TuneD profiles distributed with RHEL.

2. To see which profile is currently active, use:

   # tuned-adm active

3. Create a new directory to hold your TuneD profile:

   # mkdir /etc/tuned/<new-profile>

4. Find the system unique identifier of the selected block device:

   # udevadm info --query=property --name=/dev/device | grep -E '(WWN|SERIAL)'

   ID_WWN=0x5002538d00000000_
   ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   ID_SERIAL_SHORT=20120501030900000

   Note

   This command returns all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although using a WWN is preferred, it is not always available for every device. In such cases, you can use any value returned by the example command as the device system’s unique ID.

5. Create the /etc/tuned/my-profile/tuned.conf configuration file. In the file, set the following options:

   1. Optional: Include an existing profile:

      [main]
      include=<existing-profile>

   2. Set the selected disk scheduler for the device that matches the WWN identifier:

      [disk]
      devices_udev_regex=IDNAME=device system unique id
      elevator=selected-scheduler

      Here:

      • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).

      • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000). To match multiple devices in the devices_udev_regex option, enclose the identifiers in parentheses and separate them with vertical bars:

        devices_udev_regex=(ID_WWN=0x5002538d00000000)|(ID_WWN=0x1234567800000000)

6. Enable your profile:

   # tuned-adm profile <new-profile>

Verification

1. Verify that the TuneD profile is active and applied:

   $ tuned-adm active

   Current active profile: <profile>

   $ tuned-adm verify

   Verification succeeded, current system settings match the preset profile.
   See TuneD log file ('/var/log/tuned/tuned.log') for details.

2. Read the contents of the /sys/block/device/queue/scheduler file:

   $ cat /sys/block/device/queue/scheduler

   [mq-deadline] kyber bfq none

   In the file name, replace device with the block device name, for example sdc. The active scheduler is listed in square brackets ([]).

Procedure

1. Find the system unique identifier of the block device:

   # $ udevadm info --name=/dev/device | grep -E '(WWN|SERIAL)'

   E: ID_WWN=0x5002538d00000000
   E: ID_SERIAL=Generic-_SD_MMC_20120501030900000-0:0
   E: ID_SERIAL_SHORT=20120501030900000

   Note

   This command returns all values identified as a World Wide Name (WWN) or serial number associated with the specified block device. Although using a WWN is preferred, it is not always available for every device. In such cases, you can use any value returned by the example command as the device system’s unique ID.

2. To configure the udev rule, create the /etc/udev/rules.d/99-scheduler.rules file with the following content:

   ACTION=="add|change", SUBSYSTEM=="block", ENV{IDNAME}=="device system unique id", ATTR{queue/scheduler}="selected-scheduler"

   Here:

   • Replace IDNAME with the name of the identifier being used (for example, ID_WWN).
   • Replace device system unique id with the value of the chosen identifier (for example, 0x5002538d00000000).

3. Reload udev rules:

   # udevadm control --reload-rules

4. Apply the scheduler configuration:

   # udevadm trigger --type=devices --action=change

Procedure

1. Install the pcp package:

   # dnf install pcp

2. Enable and start the pmcd service on the host machine:

   # systemctl enable pmcd

   # systemctl start pmcd

Procedure

1. Update the pmlogger configuration:

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

2. Start the pmcd and pmlogger services:

   # systemctl start pmcd.service

   # systemctl start pmlogger.service

3. Run the required operations to record the performance data.

4. Save the output and save it to a tar.gz file named based on the host name and the current date and time:

   # cd /var/log/pcp/pmlogger/

   # tar -czf $(hostname).$(date +%F-%Hh%M).pcp.tar.gz $(hostname)

5. Extract this file and analyze the data using PCP tools.

Procedure

1. Install the pmda-openmetrics PMDA.

   # dnf -y install pcp-pmda-openmetrics

   # cd /var/lib/pcp/pmdas/openmetrics/

   # ./Install

2. Create a sample OpenMetrics file.

   # echo 'var1 {var2="var3"} 42' > /tmp/example.txt

   Replace example.txt with the file name that you want to use.

3. Verify that the file is created correctly.

   # cat /tmp/example.txt

4. Register the metric file path with the OpenMetrics agent.

   # echo "file:///tmp/example.txt" > /etc/pcp/openmetrics/example.url

5. Verify that the configuration is created correctly.

   # cat /etc/pcp/openmetrics/example.url

6. Configure systemd-tmpfiles to create the necessary symlinks.

   # echo 'L+ /var/lib/pcp/pmdas/openmetrics/config.d/example.url - - - - ../../../../../../etc/pcp/openmetrics/example.url' \ > /usr/lib/tmpfiles.d/pcp-pmda-openmetrics-cust.conf

7. Verify that the symlinks are configured correctly.

   # cat /usr/lib/tmpfiles.d/pcp-pmda-openmetrics-cust.conf

8. Create the symlinks by applying the tmpfiles configuration.

   # systemd-tmpfiles --create --remove /usr/lib/tmpfiles.d/pcp-pmda-openmetrics-cust.conf

9. Verify that the symlinks are created correctly.

   # ls -al /var/lib/pcp/pmdas/openmetrics/config.d/

10. Verify that the metric is correctly reported.

    # pminfo -f openmetrics.example.var1

    inst [0 or "0 var2:var3"] value 42

Procedure

1. Create or modify the pmlogger configuration file:

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

2. Follow pmlogconf prompts to enable or disable groups of related performance metrics and to control the logging interval for each enabled group. For example,

   # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default

   Group: per logical block device activity
   Log this group? [y]
   Logging interval? [default]

Procedure

1. Install the pcp-system-tools package:

   # dnf install pcp-system-tools

2. Configure an IP address for pmcd:

   # echo "-i 192.168.4.62" >>/etc/pcp/pmcd/pmcd.options

   Replace 192.168.4.62 with the IP address, the client should listen on. By default, pmcd is listening on the localhost.

3. Configure the firewall to add the public zone permanently:

   # firewall-cmd --permanent --zone=public --add-port=44321/tcp

   success

   # firewall-cmd --reload

   success

4. Set an SELinux boolean:

   # setsebool -P pcp_bind_all_unreserved_ports on

5. Enable the pmcd and pmlogger services:

   # systemctl enable pmcd pmlogger

   # systemctl restart pmcd pmlogger

Procedure

1. Install the pcp-system-tools package:

   # dnf install pcp-system-tools

2. Create the /etc/pcp/pmlogger/control.d/remote file with the following content:

   # DO NOT REMOVE OR EDIT THE FOLLOWING LINE
   $version=1.1
   192.168.4.13 n n PCP_ARCHIVE_DIR/rhel7u4a -r -T24h10m -c config.rhel7u4a
   192.168.4.14 n n PCP_ARCHIVE_DIR/rhel6u10a -r -T24h10m -c config.rhel6u10a
   192.168.4.62 n n PCP_ARCHIVE_DIR/rhel8u1a -r -T24h10m -c config.rhel8u1a

   Replace 192.168.4.13, 192.168.4.14 and 192.168.4.62 with the client IP addresses.

3. Enable the pmcd and pmlogger services:

   # systemctl enable pmcd pmlogger

   # systemctl restart pmcd pmlogger

Procedure

1. Install the following packages:

   1. Install the pcp-system-tools:

      # dnf install pcp-system-tools

   2. Install the pmda-postfix package to monitor postfix:

      # dnf install pcp-pmda-postfix postfix

   3. Install the logging daemon:

      # dnf install rsyslog

   4. Install the mail client for testing:

      # dnf install mutt

2. Enable the postfix and rsyslog services:

   # systemctl enable postfix rsyslog

   # systemctl restart postfix rsyslog

3. Enable the SELinux boolean, so that pmda-postfix can access the required log files:

   # setsebool -P pcp_read_generic_logs=on

4. Install the PMDA:

   # cd /var/lib/pcp/pmdas/postfix/

   # ./Install

   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Waiting for pmcd to terminate ...
   Starting pmcd ...
   Check postfix metrics have appeared ... 7 metrics and 58 values

Procedure

1. Launch the PCP Charts application from the command line:

   # pmchart

   The pmtime server settings are located at the bottom. With start and pause buttons you can control:

   • The interval in which PCP polls the metric data
   • The date and time for the metrics of historical data
2. Click File and then New Chart to select metric from both the local machine and remote machines by specifying their host name or address. Advanced configuration options include the ability to manually set the axis values for the chart, and to manually choose the color of the plots.

3. Record the views created in the PCP Charts application:

   Following are the options to take images or record the views created in the PCP Charts application:

   • Click File and then Export to save an image of the current view.
   • Click Record and then Start to start a recording.
   • Click Record and then Stop to stop the recording. After stopping the recording, the recorded metrics are archived to be viewed later.
4. Optional: In the PCP Charts application, the main configuration file, known as the view, stores the metadata associated with one or more charts. This metadata describes all chart aspects, including the metrics used and the chart columns.

5. Optional: Save the custom view configuration by clicking File and then Save View, and load the view configuration later.

   The following example of the PCP Charts application view configuration file describes a stacking chart graph showing the total number of bytes read and written to the given XFS file system loop1:

   # pmchart

   version 1
   chart title "Filesystem Throughput /loop1" style stacking antialiasing off
       plot legend Read rate   metric xfs.read_bytes   instance  loop1
       plot legend Write rate  metric xfs.write_bytes  instance  loop1

Procedure

1. Install PCP:

   # dnf install pcp-zeroconf

2. Install packages required for the pyodbc driver:

   # dnf install gcc-c++ python3-devel unixODBC-devel

   # dnf install python3-pyodbc

3. Install the mssql agent:

   1. Install the Microsoft SQL Server domain agent for PCP:

      # dnf install pcp-pmda-mssql

   2. Edit the /etc/pcp/mssql/mssql.conf file to configure the SQL server account’s username and password for the mssql agent. Ensure that the account you configure has access rights to performance data.

      username: user_name
      password: user_password

      Replace user_name with the SQL Server account username and user_password with the SQL Server user password for this account.

4. Install the agent:

   # cd /var/lib/pcp/pmdas/mssql

   # ./Install

   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD ...
   Check mssql metrics have appeared ... 168 metrics and 598 values
   [...]

Procedure

1. Install the following packages:

   # dnf install grafana grafana-pcp

2. Restart and enable grafana-server:

   # systemctl restart grafana-server

   # systemctl enable grafana-server

3. Open the server’s firewall for network traffic to the Grafana service.

   # firewall-cmd --permanent --add-service=grafana

   success

   # firewall-cmd --reload

   success

Procedure

1. Install the valkey package:

   # dnf install valkey

2. Start and enable the pmproxy and valkey services:

   # systemctl start pmproxy valkey

   # systemctl enable pmproxy valkey

3. Restart grafana-server:

   # systemctl restart grafana-server

Procedure

1. On the client system, open a browser and access the grafana-server on port 3000 by using the http://192.0.2.0:3000 link.

   Replace 192.0.2.0 with your machine IP when accessing Grafana web UI from a remote machine, or with localhost when accessing the web UI locally.

2. For the first login, enter admin in both the Email or username and Password fields.
3. Grafana prompts to set a New password to create a secured account. If you want to set it later, click Skip.
4. From the more options icon (☰) on the top left, click Administration > Plugins.
5. In the Plugins tab, type performance co-pilot in the Search by name or type text box and then click the Performance Co-Pilot (PCP) plugin.
6. In the Plugins / Performance Co-Pilot pane, click Enable.

7. Click the Grafana icon. The Grafana Home page is displayed.

   Figure 8.1. Home Dashboard

   
   Note

   The top corner of the screen has a Settings icon, but it controls the general Dashboard settings.

8. In the Grafana Home page, click Add your first data source to add Valkey, PCP bpftrace, and PCP Vector data sources.

   • To add PCP valkey data source, view default dashboard, create a panel, and an alert rule, see Creating panels and alerts in PCP valkey data source.
   • To add pcp bpftrace data source and view the default dashboard, see Viewing the PCP bpftrace System Analysis dashboard.
   • To add pcp vector data source, view the default dashboard, and to view the vector checklist, see Viewing the PCP Vector Checklist.

9. Optional: From the menu, hover over the admin profile icon to change the Preferences including Edit Profile, Change Password, or to Sign out.

   For more information, see the grafana-cli and grafana-server man pages on your system.

Procedure

1. As a root user, open the /etc/valkey/valkey.conf file and adjust the TLS/SSL options to reflect the following properties:

   port 0
   tls-port 6379
   tls-cert-file /etc/valkey/valkey.crt
   tls-key-file /etc/valkey/valkey.key
   tls-client-key-file /etc/valkey/client.key
   tls-client-cert-file /etc/valkey/client.crt
   tls-ca-cert-file /etc/valkey/ca.crt

2. Ensure valkey can access the TLS certificates:

   # su valkey -s /bin/bash -c \

     'ls -1 /etc/valkey/ca.crt /etc/valkey/valkey.key /etc/valkey/valkey.crt'
   /etc/valkey/ca.crt
   /etc/valkey/valkey.crt
   /etc/valkey/valkey.key

3. Restart the valkey server to apply the configuration changes:

   # systemctl restart valkey

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type valkey in the Filter by name or type text box and then click Valkey.

4. In the Data Sources / Valkey pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab > Import > Valkey: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 8.2. Valkey: Host overview

      

5. Add a new panel:

   1. From the menu, hover over the Create icon > Dashboard > Add new panel icon to add a panel.
   2. In the Query tab, select the Valkey from the query list instead of the selected default option and in the text field of A, enter metric, for example, kernel.all.load to visualize the kernel load graph.
   3. Optional: Add Panel title and Description, and update other options from the Settings.
   4. Click Save to apply changes and save the dashboard. Add Dashboard name.

   5. Click Apply to apply changes and go back to the dashboard.

      Figure 8.3. Valkey query panel

      

6. Create an alert rule:

   1. In the Valkey query panel, click Alert and then click Create Alert.
   2. Edit the Name, Evaluate query, and For fields from the Rule, and specify the Conditions for your alert.

   3. Click Save to apply changes and save the dashboard. Click Apply to apply changes and go back to the dashboard.

      Figure 8.4. Creating alerts in the Valkey panel

      
   4. Optional: In the same panel, scroll down and click Delete icon to delete the created rule.
   5. Optional: From the menu, click Alerting icon to view the created alert rules with different alert statuses, to edit the alert rule, or to pause the existing rule from the Alert Rules tab. To add a notification channel for the created alert rule to receive an alert notification from Grafana, see Adding notification channels for alerts.

Procedure

1. From the menu, hover over the Alerting icon > click Notification channels > Add channel.

2. In the Add notification channel details pane, perform the following:

   1. Enter your name in the Name text box.
   2. Select the communication Type, for example, Email and enter the email address. You can add multiple email addresses using the ; separator.
   3. Optional: Configure Optional Email settings and Notification settings.
3. Click Save.

4. Select a notification channel in the alert rule:

   1. From the menu, hover over the Alerting icon and then click Alert rules.
   2. From the Alert Rules tab, click the created alert rule.
   3. On the Notifications tab, select your notification channel name from the Send to option, and then add an alert message.
5. Click Apply.

Procedure

1. Install the SASL framework for the scram-sha-256 authentication mechanism:

   # dnf install cyrus-sasl-scram cyrus-sasl-lib

2. Specify the supported authentication mechanism and the user database path in the pmcd.conf file:

   # vi /etc/sasl2/pmcd.conf

   mech_list: scram-sha-256
   sasldb_path: /etc/pcp/passwd.db

3. Create a new user:

   # useradd -r metrics

   Replace metrics by your user name.

4. Add the created user in the user database:

   # saslpasswd2 -a pmcd metrics

   Password:
   Again (for verification):

   To add the created user, it is required to enter the metrics account password.

5. Set the permissions of the user database:

   # chown root:pcp /etc/pcp/passwd.db

   # chmod 640 /etc/pcp/passwd.db

6. Restart the pmcd service:

   # systemctl restart pmcd

Procedure

1. Install the pcp-pmda-bpftrace package:

   # dnf install pcp-pmda-bpftrace

2. Edit the bpftrace.conf file and add the user:

   # vi /var/lib/pcp/pmdas/bpftrace/bpftrace.conf

   [dynamic_scripts]
   enabled = true
   auth_enabled = true
   allowed_users = root,metrics

   Replace metrics by your user name from Setting up authentication between PCP components.

3. Install bpftrace PMDA:

   # cd /var/lib/pcp/pmdas/bpftrace/

   # ./Install

   Updating the Performance Metrics Name Space (PMNS) ...
   Terminate PMDA if already installed ...
   Updating the PMCD control file, and notifying PMCD …
   Check bpftrace metrics have appeared ... 7 metrics and 6 values

   The pmda-bpftrace is now installed, and can only be used after authenticating your user. For more information, see Viewing the PCP bpftrace System Analysis dashboard.

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type bpftrace in the Filter by name or type text box and then click PCP bpftrace.

4. In the Data Sources / PCP bpftrace pane, perform the following:

   1. Add http://localhost:44322 in the URL field.
   2. Toggle the Basic Auth option and add the created user credentials in the User and Password field.

   3. Click Save & Test.

      Figure 8.5. Adding PCP bpftrace in the data source

      

   4. Click Dashboards tab > Import > PCP bpftrace: System Analysis to see a dashboard with an overview of any useful metrics.

      Figure 8.6. PCP bpftrace: System Analysis

      

Procedure

1. Log into the Grafana web UI.
2. In the Grafana Home page, click Add your first data source.
3. In the Add data source pane, type vector in the Filter by name or type text box and then click PCP Vector.

4. In the Data Sources / PCP Vector pane, perform the following:

   1. Add http://localhost:44322 in the URL field and then click Save & Test.

   2. Click Dashboards tab > Import * PCP Vector: Host Overview to see a dashboard with an overview of any useful metrics.

      Figure 8.7. PCP Vector: Host Overview

      

5. From the menu, hover over the Performance Co-Pilot plugin and then click PCP Vector Checklist.

   In the PCP checklist, click help or warning icon to view the related troubleshooting or reference links.

   Figure 8.8. Performance Co-Pilot / PCP Vector Checklist

   

Procedure

1. Hover the cursor over the Dashboards tab and click + New dashboard.
2. In the Add panel menu, click Add a new panel.

3. In the Query tab:

   1. Select Valkey from the query list instead of the selected default option.
   2. In the text field of A, enter a metric, for example, kernel.all.load to visualize the kernel load graph.
4. Click the visualization dropdown menu, which is set to Time series by default, and then click Heatmap.
5. Optional: In the Panel Options dropdown menu, add a Panel Title and Description.

6. In the Heatmap dropdown menu, under the Calculate from data setting, click Yes.

   Figure 8.9. Heatmap

   
7. Optional: In the Colors dropdown menu, change the Scheme from the default Orange and select the number of steps (color shades).

8. Optional: In the Tooltip dropdown menu, under the Show histogram (Y Axis) setting, click the toggle to display a cell’s position within its specific histogram when hovering your cursor over a cell in the heatmap. For example:

   

Procedure

1. View all Grafana-related SELinux booleans:

   # semanage boolean -l | grep grafana

2. Enable a SELinux boolean:

   # setsebool [-P] <boolean_name> on

   • Replace <boolean_name> with the name of the SELinux boolean you want to enable.
   • Use the -P option to make the change persistent across system reboots.

3. Optional: Turn off any SELinux option:

   # setsebool [-P] <boolean_name> off

Procedure

1. View what driver is loaded and in what mode:

   # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver

   • intel_pstate is returned if the Intel P-State driver is loaded and in active mode.
   • intel_cpufreq is returned if the Intel P-State driver is loaded and in passive mode.
   • acpi-cpufreq is returned if the ACPI CPUfreq driver is loaded.

2. While using the Intel P-State driver:

   1. Add the following argument to the kernel boot command line to force the driver to run in passive mode:

      intel_pstate=passive

   2. To disable the Intel P-State driver and use the ACPI CPUfreq driver, add the following argument to the kernel boot command line:

      intel_pstate=disable

Procedure

1. Start GDB attached to the application or library you want to debug. GDB automatically recognizes missing debugging information and suggests a command to run.

   $ gdb -q /bin/ls

   Reading symbols from /bin/ls...Reading symbols from .gnu_debugdata for /usr/bin/ls...(no debugging symbols found)...done.
   (no debugging symbols found)...done.
   Missing separate debuginfos, use: dnf debuginfo-install coreutils-9.5-6.el10.x86_64
   (gdb)

2. To exit GDB, type q and confirm with Enter.

   (gdb) q

3. Run the command suggested by GDB to install the required debuginfo packages:

   # dnf debuginfo-install coreutils-9.5-6.el10.x86_64

   The dnf package management tool provides a summary of the changes, asks for confirmation and once you confirm, downloads and installs all the necessary files. In case GDB is not able to suggest the debuginfo package, follow the procedure described in Getting debuginfo packages for an application or library manually.

Procedure

1. Capture performance data you are interested in investigating further:

   # perf record -a --call-graph fp sleep <seconds>

   This example generates a perf.data over the entire system for a period of seconds seconds as dictated by the use of the sleep command. It would also capture call graph data using the frame pointer method.

2. Generate an archive file containing debug symbols of the recorded data:

   # perf archive

Procedure

1. Copy both the perf.data file and the archive file into your current active directory.

2. Extract the archive file into ~/.debug:

   # mkdir -p ~/.debug

   # tar xf perf.data.tar.bz2 -C ~/.debug

   Note

   The archive file might also be named perf.data.tar.gz.

3. Open the perf.data file for further analysis:

   # perf report

Procedure

1. Record performance data from specific CPUs:

   Use the -C option with perf record to target specific CPUs. The following examples demonstrate how to specify individual CPUs or a range.

   • To sample data from selected CPUs (comma-separated):

     # perf record -C 0,1 sleep <seconds>

     This command samples and records performance data from CPUs 0 and 1 for the specified number of seconds.

   • To sample data from a range of CPUs:

     # perf record -C 0-2 sleep <seconds>

     This command samples and records performance data from CPUs 0, 1, and 2 over the specified duration.

2. Analyze the recorded performance data:

   Use the perf report command to read and analyze the perf.data file.

   # perf report

   This command displays the contents of the perf.data file.

   Note

   If you want to see which CPU each sample was recorded on, see Displaying which CPU samples were taken on with perf report.

Procedure

1. View the function lines where you can place a uprobe:

   $ perf probe -x ./your_executable -L main

   <main@/home/user/my_executable:0>
                 0  int main(int argc, const char *argv) 1 { int err; const char *cmd; char sbuf[STRERR_BUFSIZE]; / libsubcmd init */
                 7         exec_cmd_init("perf", PREFIX, PERF_EXEC_PATH, EXEC_PATH_ENVIRONMENT);
   8         pager_init(PERF_PAGER_ENVIRONMENT);

2. Create the uprobe for the function line that you want to monitor:

   # perf probe -x ./my_executable main:8

   Added new event:
             probe_my_executable:main_L8   (on main:8 in /home/user/my_executable)
   
   you can now use it in all perf tools, such as:
   
   Perf record -e probe_my_executable:main_L8 -aR sleep 1

Procedure

1. Sample the memory accesses:

   # perf mem record -a sleep <seconds>

   This command samples memory accesses across all CPUs for a period of <seconds> seconds as dictated by the sleep command. You can replace the sleep command for any command during which you want to sample memory access data. By default, perf mem samples both memory loads and stores. You can select only one memory operation by using the -t option and specifying either load or store between perf mem and record. For loads, information over the memory hierarchy level, TLB memory accesses, bus snoops, and memory locks is captured.

2. Open the perf.data file for analysis:

   # perf mem report

   If you have used the example commands, the output is:

   Available samples
   35k cpu/mem-loads,ldlat=30/P
   54k cpu/mem-stores/P

   The cpu/mem-loads,ldlat=30/P line denotes data collected over memory loads and the cpu/mem-stores/P line denotes data collected over memory stores. Highlight the category of interest and press Enter to view the data:

   Samples: 35K of event 'cpu/mem-loads,ldlat=30/P', Event count (approx.): 4067062
   Overhead       Samples  Local Weight  Memory access             Symbol                                                                 Shared Object                 Data Symbol                                                     Data Object                            Snoop         TLB access              Locked
      0.07%            29  98            L1 or L1 hit              [.] 0x000000000000a255                                                 libspeexdsp.so.1.5.0          [.] 0x00007f697a3cd0f0                                          anon                                   None          L1 or L2 hit            No
      0.06%            26  97            L1 or L1 hit              [.] 0x000000000000a255                                                 libspeexdsp.so.1.5.0          [.] 0x00007f697a3cd0f0                                          anon                                   None          L1 or L2 hit            No
      0.06%            25  96            L1 or L1 hit              [.] 0x000000000000a255                                                 libspeexdsp.so.1.5.0          [.] 0x00007f697a3cd0f0                                          anon                                   None          L1 or L2 hit            No
      0.06%             1  2325          Uncached or N/A hit       [k] pci_azx_readl                                                      [kernel.kallsyms]             [k] 0xffffb092c06e9084                                          [kernel.kallsyms]                      None          L1 or L2 hit            No
      0.06%             1  2247          Uncached or N/A hit       [k] pci_azx_readl                                                      [kernel.kallsyms]             [k] 0xffffb092c06e8164                                          [kernel.kallsyms]                      None          L1 or L2 hit            No
      0.05%             1  2166          L1 or L1 hit              [.] 0x00000000038140d6                                                 libxul.so                     [.] 0x00007ffd7b84b4a8                                          [stack]                                None          L1 or L2 hit            No
      0.05%             1  2117          Uncached or N/A hit       [k] check_for_unclaimed_mmio                                           [kernel.kallsyms]             [k] 0xffffb092c1842300                                          [kernel.kallsyms]                      None          L1 or L2 hit            No
      0.05%            22  95            L1 or L1 hit              [.] 0x000000000000a255                                                 libspeexdsp.so.1.5.0          [.] 0x00007f697a3cd0f0                                          anon                                   None          L1 or L2 hit            No
      0.05%             1  1898          L1 or L1 hit              [.] 0x0000000002a30e07                                                 libxul.so                     [.] 0x00007f610422e0e0                                          anon                                   None          L1 or L2 hit            No
      0.05%             1  1878          Uncached or N/A hit       [k] pci_azx_readl                                                      [kernel.kallsyms]             [k] 0xffffb092c06e8164                                          [kernel.kallsyms]                      None          L2 miss                 No
      0.04%            18  94            L1 or L1 hit              [.] 0x000000000000a255                                                 libspeexdsp.so.1.5.0          [.] 0x00007f697a3cd0f0                                          anon                                   None          L1 or L2 hit            No
      0.04%             1  1593          Local RAM or RAM hit      [.] 0x00000000026f907d                                                 libxul.so                     [.] 0x00007f3336d50a80                                          anon                                   Hit           L2 miss                 No
      0.03%             1  1399          L1 or L1 hit              [.] 0x00000000037cb5f1                                                 libxul.so                     [.] 0x00007fbe81ef5d78                                          libxul.so                              None          L1 or L2 hit            No
      0.03%             1  1229          LFB or LFB hit            [.] 0x0000000002962aad                                                 libxul.so                     [.] 0x00007fb6f1be2b28                                          anon                                   None          L2 miss                 No
      0.03%             1  1202          LFB or LFB hit            [.] __pthread_mutex_lock                                               libpthread-2.29.so            [.] 0x00007fb75583ef20                                          anon                                   None          L1 or L2 hit            No
      0.03%             1  1193          Uncached or N/A hit       [k] pci_azx_readl                                                      [kernel.kallsyms]             [k] 0xffffb092c06e9164                                          [kernel.kallsyms]                      None          L2 miss                 No
   0.03%             1  1191          L1 or L1 hit              [k] azx_get_delay_from_lpib                                            [kernel.kallsyms]             [k] 0xffffb092ca7efcf0                                          [kernel.kallsyms]                      None          L1 or L2 hit            No

3. Optional: Sort your results to investigate different aspects of interest when displaying the data. For example, to sort data over memory loads by type of memory accesses occurring during the sampling period in descending order of overhead they account for:

   # perf mem -t load report --sort=mem

   For example, the output can be:

   Samples: 35K of event 'cpu/mem-loads,ldlat=30/P', Event count (approx.): 40670
   Overhead       Samples  Memory access
     31.53%          9725  LFB or LFB hit
     29.70%         12201  L1 or L1 hit
     23.03%          9725  L3 or L3 hit
     12.91%          2316  Local RAM or RAM hit
      2.37%           743  L2 or L2 hit
      0.34%             9  Uncached or N/A hit
      0.10%            69  I/O or N/A hit
      0.02%           825  L3 miss

Procedure

1. Open the perf.data file for analysis:

   # perf c2c report --stdio

   This command visualizes the perf.data file into several graphs within the command line:

   =================================================
              Trace Event Information
   =================================================
    Total records                     :     329219
    Locked Load/Store Operations      :      14654
    Load Operations                   :      69679
    Loads - uncacheable               :          0
    Loads - IO                        :          0
    Loads - Miss                      :       3972
    Loads - no mapping                :          0
    Load Fill Buffer Hit              :      11958
    Load L1D hit                      :      17235
    Load L2D hit                      :         21
    Load LLC hit                      :      14219
    Load Local HITM                   :       3402
    Load Remote HITM                  :      12757
    Load Remote HIT                   :       5295
    Load Local DRAM                   :        976
    Load Remote DRAM                  :       3246
    Load MESI State Exclusive         :       4222
    Load MESI State Shared            :          0
    Load LLC Misses                   :      22274
    LLC Misses to Local DRAM          :        4.4%
    LLC Misses to Remote DRAM         :       14.6%
    LLC Misses to Remote cache (HIT)  :       23.8%
    LLC Misses to Remote cache (HITM) :       57.3%
    Store Operations                  :     259539
    Store - uncacheable               :          0
    Store - no mapping                :         11
    Store L1D Hit                     :     256696
    Store L1D Miss                    :       2832
    No Page Map Rejects               :       2376
    Unable to parse data source       :          1
   
   =================================================
      Global Shared Cache Line Event Information
   =================================================
    Total Shared Cache Lines          :         55
    Load HITs on shared lines         :      55454
    Fill Buffer Hits on shared lines  :      10635
    L1D hits on shared lines          :      16415
    L2D hits on shared lines          :          0
    LLC hits on shared lines          :       8501
    Locked Access on shared lines     :      14351
    Store HITs on shared lines        :     109953
    Store L1D hits on shared lines    :     109449
    Total Merged records              :     126112
   
   =================================================
                    c2c details
   =================================================
    Events                            : cpu/mem-loads,ldlat=30/P
   	                                    : cpu/mem-stores/P
    Cachelines sort on                : Remote HITMs
    Cacheline data grouping          : offset,pid,iaddr
   
   =================================================
   	   Shared Data Cache Line Table
   =================================================
   #
   #                              Total      Rmt  ----- LLC Load Hitm -----  ---- Store Reference ----  --- Load Dram ----      LLC    Total  ----- Core Load Hit -----  -- LLC Load Hit --
   # Index           Cacheline  records     Hitm    Total      Lcl      Rmt    Total    L1Hit   L1Miss       Lcl       Rmt  Ld Miss    Loads       FB       L1       L2       Llc       Rmt
   # .....  ..................  .......  .......  .......  .......  .......  .......  .......  .......  ........  ........  .......  .......  .......  .......  .......  ........  ........
   #
         0            0x602180   149904   77.09%    12103     2269     9834   109504   109036      468       727      2657    13747    40400     5355    16154        0      2875       529
         1            0x602100    12128   22.20%     3951     1119     2832        0        0        0        65       200     3749    12128     5096      108        0      2056       652
         2  0xffff883ffb6a7e80      260    0.09%       15        3       12      161      161        0         1         1       15       99       25       50        0         6         1
         3  0xffffffff81aec000      157    0.07%        9        0        9        1        0        1         0         7       20      156       50       59        0        27         4
         4  0xffffffff81e3f540      179    0.06%        9        1        8      117       97       20         0        10       25       62       11        1        0        24         7
   
   =================================================
         Shared Cache Line Distribution Pareto
   =================================================
   #
   #        ----- HITM -----  -- Store Refs --        Data address                               ---------- cycles ----------       cpu                                     Shared
   #   Num      Rmt      Lcl   L1 Hit  L1 Miss              Offset      Pid        Code address  rmt hitm  lcl hitm      load       cnt               Symbol                Object                  Source:Line  Node{cpu list}
   # .....  .......  .......  .......  .......  ..................  .......  ..................  ........  ........  ........  ........  ...................  ....................  ...........................  ....
   #
     -------------------------------------------------------------
         0     9834     2269   109036      468            0x602180
     -------------------------------------------------------------
             65.51%   55.88%   75.20%    0.00%                 0x0    14604            0x400b4f     27161     26039     26017         9  [.] read_write_func  no_false_sharing.exe  false_sharing_example.c:144   0{0-1,4}  1{24-25,120}  2{48,54}  3{169}
   	   0.41%    0.35%    0.00%    0.00%                 0x0    14604            0x400b56     18088     12601     26671         9  [.] read_write_func  no_false_sharing.exe  false_sharing_example.c:145   0{0-1,4}  1{24-25,120}  2{48,54}  3{169}
   	   0.00%    0.00%   24.80%  100.00%                 0x0    14604            0x400b61         0         0         0         9  [.] read_write_func  no_false_sharing.exe  false_sharing_example.c:145   0{0-1,4}  1{24-25,120}  2{48,54}  3{169}
   	   7.50%    9.92%    0.00%    0.00%                0x20    14604            0x400ba7      2470      1729      1897         2  [.] read_write_func  no_false_sharing.exe  false_sharing_example.c:154   1{122}  2{144}
   	  17.61%   20.89%    0.00%    0.00%                0x28    14604            0x400bc1      2294      1575      1649         2  [.] read_write_func  no_false_sharing.exe  false_sharing_example.c:158   2{53}  3{170}
   	   8.97%   12.96%    0.00%    0.00%                0x30    14604            0x400bdb      2325      1897      1828         2  [.] read_write_func  no_false_sharing.exe  false_sharing_example.c:162   0{96}  3{171}
   
     -------------------------------------------------------------
         1     2832     1119        0        0            0x602100
     -------------------------------------------------------------
   	  29.13%   36.19%    0.00%    0.00%                0x20    14604            0x400bb3      1964      1230      1788         2  [.] read_write_func  no_false_sharing.exe  false_sharing_example.c:155   1{122}  2{144}
   	  43.68%   34.41%    0.00%    0.00%                0x28    14604            0x400bcd      2274      1566      1793         2  [.] read_write_func  no_false_sharing.exe  false_sharing_example.c:159   2{53}  3{170}
   27.19%   29.40%    0.00%    0.00%                0x30    14604            0x400be7      2045      1247      2011         2  [.] read_write_func  no_false_sharing.exe  false_sharing_example.c:163   0{96}  3{171}

Procedure

1. Open the perf.data file:

   # perf c2c report --stdio

   This opens the perf.data file in the command line.

2. In the Trace Event Information table, locate the row containing the values for LLC Misses to Remote Cache (HITM):

   The percentage in the value column of the LLC Misses to Remote Cache (HITM) row represents the percentage of LLC misses that were occurring across NUMA nodes in modified cache-lines and is a key indicator that false sharing has occurred.

   =================================================
               Trace Event Information
   =================================================
     Total records                     :     329219
     Locked Load/Store Operations      :      14654
     Load Operations                   :      69679
     Loads - uncacheable               :          0
     Loads - IO                        :          0
     Loads - Miss                      :       3972
     Loads - no mapping                :          0
     Load Fill Buffer Hit              :      11958
     Load L1D hit                      :      17235
     Load L2D hit                      :         21
     Load LLC hit                      :      14219
     Load Local HITM                   :       3402
     Load Remote HITM                  :      12757
     Load Remote HIT                   :       5295
     Load Local DRAM                   :        976
     Load Remote DRAM                  :       3246
     Load MESI State Exclusive         :       4222
     Load MESI State Shared            :          0
     Load LLC Misses                   :      22274
     LLC Misses to Local DRAM          :        4.4%
     LLC Misses to Remote DRAM         :       14.6%
     LLC Misses to Remote cache (HIT)  :       23.8%
     LLC Misses to Remote cache (HITM) : 57.3%
     Store Operations                  :     259539
     Store - uncacheable               :          0
     Store - no mapping                :         11
     Store L1D Hit                     :     256696
     Store L1D Miss                    :       2832
     No Page Map Rejects               :       2376
     Unable to parse data source       :          1

3. Inspect the Rmt column of the LLC Load Hitm field of the Shared Data Cache Line Table:

   =================================================
                Shared Data Cache Line Table
   =================================================
     #
     #                              Total      Rmt  ----- LLC Load Hitm -----  ---- Store Reference ----  --- Load Dram ----      LLC    Total  ----- Core Load Hit -----  -- LLC Load Hit --
     # Index           Cacheline  records     Hitm    Total      Lcl      Rmt    Total    L1Hit   L1Miss       Lcl       Rmt  Ld Miss    Loads       FB       L1       L2       Llc       Rmt
     # .....  ..................  .......  .......  .......  .......  .......  .......  .......  .......  ........  ........  .......  .......  .......  .......  .......  ........  ........
     #
           0            0x602180   149904   77.09%    12103     2269     9834   109504   109036      468       727      2657    13747    40400     5355    16154        0      2875       529
           1            0x602100    12128   22.20%     3951     1119     2832        0        0        0        65       200     3749    12128     5096      108        0      2056       652
           2  0xffff883ffb6a7e80      260    0.09%       15        3       12      161      161        0         1         1       15       99       25       50        0         6         1
           3  0xffffffff81aec000      157    0.07%        9        0        9        1        0        1         0         7       20      156       50       59        0        27         4
           4  0xffffffff81e3f540      179    0.06%        9        1        8      117       97       20         0        10       25       62       11        1        0        24         7

   The table is sorted in descending order by the amount of remote Hitm detected per cache line. A high number in the Rmt column of the LLC Load Hitm section indicates false sharing. Further inspection of the cache line where it occurred is required to debug the false sharing activity.

Procedure

1. Configure the control pipe interface:

   # mkfifo control ack perf.pipe

2. Run perf record with the control file configuration and events you are interested in enabling:

   # perf record --control=fifo:control,ack -D -1 --no-buffering -e 'sched:*' -o - > perf.pipe

   In this example, declaring 'sched:*' after the -e option starts perf record with scheduler events.

3. In a second command line, start the read side of the control pipe:

   # cat perf.pipe | perf --no-pager script -i -

4. Starting the read side of the control pipe triggers the following message in the first command line:

   Events disabled

5. In a third command line, enable a tracepoint by using the control file:

   # echo 'enable sched:sched_process_fork' > control

   This command triggers perf to scan the current event list in the control file for the declared event. If the event is present, the tracepoint is enabled and the following message appears in the first command line:

   event sched:sched_process_fork enabled

6. Once the tracepoint is enabled, the second command line displays the output from perf detecting the tracepoint:

   bash 33349 [034] 149587.674295: sched:sched_process_fork: comm=bash pid=33349 child_comm=bash child_pid=34056

Procedure

1. Log in to the RHEL 10 web console.
2. Click Overview.

3. In the Configuration section, click the current performance profile.

   

4. In the Change performance profile dialog box, set the required profile.
5. Click Change profile.

Procedure

1. Log in to the RHEL 10 web console.
2. Click Overview.

3. In the Usage section, click View metrics and history.

   

   The Metrics and history section opens and displays: the current system configuration and usage the performance metrics in a graphical form over a user-specified time interval

Procedure

1. Log in to the RHEL 10 web console.
2. In the Overview page, click View metrics and history in the Usage table.
3. Click the Metrics settings button.

4. Move the Export to network slider to the active position.

   If you do not have the valkey package installed, the web console prompts you to install it.

5. To open the pmproxy service, select a zone from a drop-down list and click the Add pmproxy button.
6. Click Save.

Verification

1. Click Networking.
2. In the Firewall table, click the Edit rules and zones button.

3. Search for pmproxy in your selected zone.

   Important

   Repeat this procedure on all systems you want to monitor for performance.

Procedure

1. Run the execsnoop program in one command line:

   # /usr/share/bcc/tools/execsnoop

2. To create a short-lived process of the ls command, in another command line, enter:

   $ ls /usr/share/bcc/tools/doc/

   The command line running execsnoop shows the following output:

   PCOMM	PID    PPID   RET ARGS
   ls   	8382   8287     0 /usr/bin/ls --color=auto /usr/share/bcc/tools/doc/

   The execsnoop program prints a line of output for each new process that consumes system resources. It even detects processes of programs that run very shortly, such as ls, and most monitoring tools would not register them. The execsnoop output displays the following fields:

   PCOMM

   The parent process name. (ls)

   PID

   The process ID. (8382)

   PPID

   The parent process ID. (8287)

   RET

   The return value of the exec() system call (0), which loads program code into new processes.

   ARGS

   The location of the started program with arguments.

   For more information, see the /usr/share/bcc/tools/doc/execsnoop_example.txt file and the exec(3) man page on your system.

Procedure

1. In one command line, run the opensnoop program to print the output for files opened only by the process of the uname command:

   # /usr/share/bcc/tools/opensnoop -n uname

2. In another command line, enter the command to open certain files:

   $ uname

   The command line running opensnoop shows the output similar to the following:
   PID    COMM 	FD ERR PATH
   8596   uname 	3  0   /etc/ld.so.cache
   8596   uname 	3  0   /lib64/libc.so.6
   8596   uname 	3  0   /usr/lib/locale/locale-archive
   ...

   The opensnoop program watches the open() system call across the whole system, and prints a line of output for each file that uname tries to open along the way. The opensnoop output displays the following fields:

   PID

   The process ID. (8596)

   COMM

   The process name. (uname)

   FD

   The file descriptor - a value that open() returns to refer to the open file. (3)

   ERR

   Any errors.

   PATH

   The location of files that open() tries to open.

   If a command tries to read a non-existent file, the FD column returns -1. Also, the ERR column prints a value corresponding to the relevant error. Use opensnoop to identify an application that does not behave properly. For more information, see the /usr/share/bcc/tools/doc/opensnoop_example.txt file and open(2) man page on your system.

Procedure

1. Run the biotop program in one command line with 30 as an argument to produce 30 second summary:

   # /usr/share/bcc/tools/biotop 30

   When you do not provide any argument, the output screen refreshes every 1 second by default.

2. In another command line, enter command to read the content from the local hard disk device and write the output to the /dev/zero file:

   # dd if=/dev/vda of=/dev/zero

   This step generates certain I/O traffic to illustrate biotop. The command line running biotop shows an output similar to the following:

   PID    COMM             D MAJ MIN DISK       I/O  Kbytes     AVGms
   9568   dd               R 252 0   vda      16294 14440636.0  3.69
   48     kswapd0          W 252 0   vda       1763 120696.0    1.65
   7571   gnome-shell      R 252 0   vda        834 83612.0     0.33
   1891   gnome-shell      R 252 0   vda       1379 19792.0     0.15
   7515   Xorg             R 252 0   vda        280  9940.0     0.28
   7579   llvmpipe-1       R 252 0   vda        228  6928.0     0.19
   9515   gnome-control-c  R 252 0   vda         62  6444.0     0.43
   8112   gnome-terminal-  R 252 0   vda         67  2572.0     1.54
   7807   gnome-software   R 252 0   vda         31  2336.0     0.73
   9578   awk              R 252 0   vda         17  2228.0     0.66
   7578   llvmpipe-0       R 252 0   vda        156  2204.0     0.07
   9581   pgrep            R 252 0   vda         58  1748.0     0.42
   7531   InputThread      R 252 0   vda         30  1200.0     0.48
   7504   gdbus            R 252 0   vda          3  1164.0     0.30
   1983   llvmpipe-1       R 252 0   vda         39   724.0     0.08
   1982   llvmpipe-0       R 252 0   vda         36   652.0     0.06
   ...

   The biotop output displays the following fields:

   PID

   The process ID. (9568)

   COMM

   The process name. (dd)

   DISK

   The disk performs the read operations. (vda)

   I/O

   The number of read operations performed. (16294)

   Kbytes

   The amount of Kbytes reached by the read operations. (14,440,636)

   AVGms

   The average I/O time of read operations. (3.69)

   For more information, see the /usr/share/bcc/tools/doc/biotop_example.txt file and the dd(1) man page on your system.