Red Hat SummitChapter 4. Configuring Performance Co-Pilot
Performance Co-Pilot (PCP) is a suite of tools, services, and libraries for managing and measuring system-level performance. You use Python, Perl, C++, and C interfaces to add performance metrics.
Analysis tools use Python, C++, and C client APIs directly. Web applications can access performance data through a JSON interface. To analyze data patterns,compare live results with archived data.
- Features of PCP
- Light-weight distributed architecture useful during the centralized analysis of complex systems.
- Ability to monitor and manage real-time data.
- Ability to log and retrieve historical data.
- PCP has the following components
- The Performance Metric Collector Daemon (pmcd) collects performance data from the installed Performance Metric Domain Agents (PMDA). PMDAs can be individually loaded or unloaded on the system and are controlled by the PMCD on the same host.
-
Various client tools, such as
pminfoorpmstat, can retrieve, display, archive, and process this data on the same host or over the network. -
The
pcpandpcp-system-toolspackages provide the command-line tools and core functionality. -
The
pcp-guipackage provides the graphical application pmchart. -
The
grafana-pcppackage provides powerful web-based visualizations and alerting with Grafana.
4.1. Installing and enabling PCP
Install the required packages and enable the PCP monitoring services to start using it. You can also automate the PCP installation by using the pcp-zeroconf package. For more information about installing PCP by using pcp-zeroconf, see Configuring PCP with pcp-zeroconf.
Procedure
Install the pcp package:
# dnf install pcpEnable and start the pmcd service on the host machine:
# systemctl enable pmcd# systemctl start pmcd
Verification
Verify if the PMCD process is running on the host:
# pcpPerformance Co-Pilot configuration on arm10.local: platform: Linux arm10.local 6.12.0-55.13.1.el10_0.aarch64 #1 SMP PREEMPT_DYNAMIC Mon May 19 07:29:57 UTC 2025 aarch64 hardware: 4 cpus, 1 disk, 1 node, 3579MB RAM timezone: JST-9 services: pmcd pmcd: Version 6.3.7-1, 12 agents, 6 clients pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm jbd2 dm openmetrics
4.2. PCP archive rotation, compression, and comparison
PCP stores collected metrics in archive files that you can use for historical performance analysis. To efficiently manage storage and simplify long-term data retention, PCP rotates and compresses archives on a daily schedule. Compare archived performance data from different time periods to identify changes in system behavior.
PCP rotates local archives daily by using the pmlogger_daily utility, which is typically started shortly after midnight by the pmlogger_daily.timer systemd unit. During daily processing, PCP rotates active archives, optionally merges completed daily archives, and compresses older archive volumes to reduce disk space usage. The pmlogcompress utility performs archive compression according to the settings in the pmlogger control configuration.
When pmlogger runs as a service, archive files are created with a base name in the YYYYMMDD.HH.MM format, for example, 20260527.00.10. After daily processing, archive volumes can include additional suffixes such as .bz2 or .xz, indicating that the files have been compressed.
In addition to archive maintenance, PCP provides tools for analyzing historical performance data. The pmdiff utility compares average metric values between two archives and highlights metrics whose values differ significantly. This comparison is useful for identifying performance regressions, workload changes, or resource utilization differences between days. By default, pmdiff reports metrics whose averages have at least doubled or halved relative to each other, although you can adjust this threshold when performing comparisons.
Together, archive rotation, compression, and comparison capabilities help maintain manageable archive storage while enabling efficient analysis of historical system performance data.
Additional resources
4.3. Deploying minimal configuration for archiving collected metrics
You can configure PCP to archive performance metrics collected on the local system by using a minimal pmlogger configuration. The system stores these archives under /var/log/pcp/pmlogger/. You can analyze archives with PCP tools such as pmlogsummary and pmdiff.
Prerequisites
- PCP is installed. For more information, see Installing and enabling PCP.
Procedure
Update the
pmloggerconfiguration by reprobing the default metric groups against the localpmcdinstance:# pmlogconf -r /var/lib/pcp/config/pmlogger/config.defaultpmlogconfmight prompt to enable or disable groups of related metrics. Accept the defaults for a minimal configuration, or enable only the groups you need.Start or restart the
pmloggerservice to apply the updated configuration:# systemctl restart pmlogger.serviceOptional: If
pmloggeris not running yet, start it.# systemctl start pmlogger.serviceVerify that
pmloggeris writing archives. List the archive directory for the local host:$ ls -l /var/log/pcp/pmlogger/$(hostname)/Example output:
-rw-r--r--. 1 pcp pcp 1899948 May 27 12:55 20260527.11.59.0 -rw-r--r--. 1 pcp pcp 1640 May 27 12:54 20260527.11.59.index -rw-r--r--. 1 pcp pcp 339929 May 27 12:54 20260527.11.59.meta lrwxrwxrwx. 1 pcp pcp 375 May 27 11:59 Latest -rw-r--r--. 1 pcp pcp 27452 May 27 11:59 pmlogger.logThe archive base name is the shared prefix of the
.index,.meta, and volume files (for example,20260527.11.59). Pass this path, without a file suffix, to PCP archive tools.Summarize metrics from the current archive to confirm that data is being recorded. Use the archive base name from the earlier step:
$ pmlogsummary /var/log/pcp/pmlogger/$(hostname)/20260527.11.59 -a disk.dev.read disk.dev.writeExample output:
Log Label (Log Format Version 3) Performance metrics from host hostname.example.com commencing Tue May 27 11:59:00.000 2026 ending Tue May 27 12:55:00.000 2026 disk.dev.read ["sda"] 125.4 124.8 0.0 0.0 3300 [] 0 [] 0 count / sec disk.dev.write ["sda"] 42.1 41.9 0.0 0.0 3300 [] 0 [] 0 count / secIf
pmlogsummaryreports metrics with non-zero averages over the archive interval,pmloggeris archiving performance data successfully.
4.4. System services and tools distributed with PCP
The basic package pcp includes the system services and basic tools. You can install additional tools that are provided with the pcp-system-tools, pcp-gui, and pcp-devel packages.
Roles of system services distributed with PCP
pmcd- The Performance Metric Collector Daemon.
pmie- The Performance Metrics Inference Engine.
pmlogger- The performance metrics logger.
pmproxy- The realtime and historical performance metrics proxy, time series query and REST API service.
Tools distributed with base PCP package
pcp- Displays the current status of a Performance Co-Pilot installation.
pcp-check-
Activates, or deactivates core and optional components, such as
pmcd,pmlogger,pmproxy, and PMDAs. pcp-vmstat- Provides a high-level system performance overview every 5 seconds. Displays information about processes, memory, paging, block IO, traps, and CPU activity.
pmconfig- Displays the values of configuration parameters.
pmdiff- Compares average metric values across two archives within a time window to identify potential performance regressions.
pmdumplog- Displays control, metadata, index, and state information from a Performance Co-Pilot archive file.
pmfind- Finds PCP services on the network.
pmie- An inference engine that periodically evaluates a set of arithmetic, logical, and rule expressions. The metrics are collected either from a live system, or from a Performance Co-Pilot archive file.
pmieconf- Displays or sets configurable pmie variables.
pmiectl- Manages non-primary instances of pmie.
pminfo- Displays information about performance metrics. The metrics are collected either from a live system, or from a Performance Co-Pilot archive file.
pmlc- Interactively configures active pmlogger instances.
pmlogcheck- Identifies invalid data in a Performance Co-Pilot archive file.
pmlogconf- Creates and modifies a pmlogger configuration file.
pmlogctl- Manages non-primary instances of pmlogger.
pmloglabel- Verifies, modifies, or repairs the label of a Performance Co-Pilot archive file.
pmlogsummary- Calculates statistical information about performance metrics stored in a Performance Co-Pilot archive file.
pmprobe- Determines the availability of performance metrics.
pmsocks- Provides access to a Performance Co-Pilot hosted through a firewall.
pmstat- Periodically displays a brief summary of system performance.
pmstore- Modifies the values of performance metrics.
pmseries- Fast, scalable time series querying, using the facilities of PCP and a distributed key-value data store such as Valkey.
pmtrace- Provides a command-line interface to the trace PMDA.
pmval- An updating display of the current value of any performance metric.
Tools distributed with the separately installed pcp-system-tools package
pcp-atop- Shows the system-level occupation of the most critical hardware resources from the performance point of view: CPU, memory, disk, and network.
pcp-atopsar- Generates a system-level activity report over a variety of system resource utilization. The report is generated from a raw logfile previously recorded using pmlogger or the -w option of pcp-atop.
pcp-dmcache- Displays information about configured Device Mapper Cache targets. Metrics include device IOPS, utilization, and read/write hit/miss rates for each cache device.
pcp-dstat- Displays metrics of one system at a time. To display metrics of multiple systems, use --host option.
pcp-free- Reports on free and used memory in a system.
pcp-htop-
Lists all processes running on a system along with their command-line arguments. It is similar to the
topcommand, with vertical and horizontal scrolling and mouse interaction. pcp-ipcs- Displays information about the inter-process communication (IPC) facilities that the calling process has read access for.
pcp-mpstat- Reports CPU and interrupt-related statistics.
pcp-numastat- Displays NUMA allocation statistics from the kernel memory allocator.
pcp-pidstat- Displays information about individual tasks or processes running on the system. This includes CPU percentage, memory and stack usage, scheduling, and priority. Reports live data for the local host by default.
pcp-shping- Samples and reports on the shell-ping service metrics exported by the pmdashping Performance Metrics Domain Agent (PMDA).
pcp-ss- Displays socket statistics collected by the pmdasockets PMDA.
pcp-tapestat- Reports I/O statistics for tape devices.
pcp-uptime- Displays the system uptime, currently logged on users, and system load averages for the past 1, 5, and 15 minutes.
pcp-verify- Inspects various aspects of a Performance Co-Pilot collector’s configuration and ensures it is optimized for specific operations.
pcp-iostat- Reports I/O statistics for SCSI devices (by default) or device-mapper devices (with the -x device-mapper option).
pmrep- Reports on selected, easily customizable, performance metrics values.
Tools distributed with the separately installed pcp-gui package
pmchart- Plots performance metrics values available through the facilities of the PCP.
pmdumptext- Outputs the values of performance metrics collected live or from a PCP archive.
Tools distributed with the separately installed pcp-devel package
pmclient- Displays high-level system performance metrics by using the Performance Metrics Application Programming Interface (PMAPI).
pmdbg- Displays available Performance Co-Pilot debug control flags and their values.
pmerr- Displays available Performance Co-Pilot error codes and their corresponding error messages.
pcp-xsos- Gives a fast summary report for a system by using a single sample taken from either a PCP archive or live metrics from the system.
Other tools distributed as a separate packages
pcp-geolocate- Discovers collector system geographical labels.
pcp2openmetrics- A customizable performance metrics exporter tool from PCP to Open Metrics format. You can select any live or archived PCP metric for exporting by using either command line arguments or a configuration file.
4.5. PCP deployment architectures
PCP supports multiple deployment architectures, based on the scale of the PCP deployment, and offers many options to accomplish advanced configurations.
Available scaling deployment configuration variants, determined by sizing factors and configuration options, include the following:
- Localhost
- Each service runs locally on the monitored machine. Starting a service without any configuration changes results in a default standalone deployment on the localhost. This configuration does not support scaling beyond a single node. However, Valkey can also run in a highly available and scalable clustered mode, where data is shared across multiple hosts. You can also deploy a Valkey cluster in the cloud or use a managed Valkey cluster from a cloud provider.
- Decentralized
- The only difference between localhost and decentralized configuration is the centralized Valkey service. In this model, the host runs pmlogger service on each monitored host and retrieves metrics from a local pmcd instance. A local pmproxy service then exports the performance metrics to a central Valkey instance.
Figure 4.1. Decentralized logging
- Centralized logging - pmlogger farm
-
For resource-constrained hosts, use a
pmloggerfarm. This deployment is also called centralized logging. A single logger host runspmloggerprocesses to collect metrics from multiple remote hosts. The centralized logger host runs thepmproxyservice, which discovers the resulting PCP archives and loads the metric data into a Valkey instance.
Figure 4.2. Centralized logging - pmlogger farm
- Federated - multiple pmlogger farms
- For large scale deployments, deploy multiple pmlogger farms in a federated fashion. For example, one pmlogger farm per rack or data center. Each pmlogger farm loads the metrics into a central Valkey instance.
Figure 4.3. Federated - multiple pmlogger farms
By default, the deployment configuration for Valkey is standalone, localhost. However, Valkey can optionally perform in a highly-available and highly scalable clustered fashion, where data is shared across multiple hosts. Another option is to deploy a Valkey cluster in the cloud, or to use a managed Valkey cluster from a cloud provider.
4.6. Factors affecting scaling in PCP logging
The key factors influencing Performance Co-Pilot (PCP) logging are hardware resources, logged metrics, logging intervals, and post-upgrade archive management.
- Remote system size
-
Remote hardware such as CPUs, disks, and network interfaces, determines the volume of data collected by each
pmloggerinstance. - Logged metrics
-
The number and types of logged metrics significantly affect storage requirements. In particular, the
per-process proc.*metrics require a large amount of disk space. For example, a standardpcp-zeroconflogging at 10-second intervals uses 11 MB. Enabling proc metrics increases this to 155 MB. Additionally, the number of CPUs, block devices, and network interfaces impacts storage capacity requirements. - Logging interval
-
The frequency of metric logging determines storage usage. The expected daily PCP archive file sizes for each
pmloggerinstance are recorded in thepmlogger.logfile. PCP archives typically compress at 10:1. Use this ratio to estimate long-term disk space requirements for uncompressed data. - Managing archive updates with
pmlogrewrite -
After PCP upgrades,
pmlogrewriteupdates existing archives if changes are detected in the metric metadata between versions. The time required for this process scales linearly with the number of stored archives.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}