Red Hat SummitEste conteúdo não está disponível no idioma selecionado.
Chapter 4. Setting up PCP
Performance Co-Pilot (PCP) is a suite of tools, services, and libraries for managing and measuring system-level performance. You use Python, Perl, C++, and C interfaces to add performance metrics. Analysis tools use Python, C++, and C client APIs directly. Web applications can access performance data through a JSON interface. To analyze data patterns,compare live results with archived data.
- Features of PCP
- Light-weight distributed architecture useful during the centralized analysis of complex systems.
- Ability to monitor and manage real-time data.
- Ability to log and retrieve historical data.
- PCP has the following components
- The Performance Metric Collector Daemon (pmcd) collects performance data from the installed Performance Metric Domain Agents (PMDA). PMDAs can be individually loaded or unloaded on the system and are controlled by the PMCD on the same host.
-
Various client tools, such as
pminfoorpmstat, can retrieve, display, archive, and process this data on the same host or over the network. -
The
pcpandpcp-system-toolspackages provide the command-line tools and core functionality. -
The
pcp-guipackage provides the graphical application pmchart. -
The
grafana-pcppackage provides powerful web-based visualizations and alerting with Grafana.
4.1. Installing and enabling PCP
Install the required packages and enable the PCP monitoring services to start using it. You can also automate the PCP installation by using the pcp-zeroconf package. For more information about installing PCP by using pcp-zeroconf, see Setting up PCP with pcp-zeroconf.
Procedure
Install the pcp package:
# dnf install pcpEnable and start the pmcd service on the host machine:
# systemctl enable pmcd# systemctl start pmcd
Verification
Verify if the PMCD process is running on the host:
# pcpPerformance Co-Pilot configuration on arm10.local: platform: Linux arm10.local 6.12.0-55.13.1.el10_0.aarch64 #1 SMP PREEMPT_DYNAMIC Mon May 19 07:29:57 UTC 2025 aarch64 hardware: 4 cpus, 1 disk, 1 node, 3579MB RAM timezone: JST-9 services: pmcd pmcd: Version 6.3.7-1, 12 agents, 6 clients pmda: root pmcd proc pmproxy xfs linux nfsclient mmv kvm jbd2 dm openmetrics
4.2. Deploying a minimal PCP configuration
The minimal PCP configuration collects performance statistics on Red Hat Enterprise Linux. The configuration involves adding the minimum number of packages on a production system needed to gather data for further analysis.
Analyze the resulting tar.gz file and the archive of the pmlogger output by using various PCP tools. Then you can compare them with other performance data sources.
Prerequisites
- PCP is installed. For more information, see Installing and enabling PCP.
Procedure
Update the pmlogger configuration:
# pmlogconf -r /var/lib/pcp/config/pmlogger/config.defaultStart the pmcd and pmlogger services:
# systemctl start pmcd.service# systemctl start pmlogger.service- Run the required operations to record the performance data.
Save the output and save it to a tar.gz file named based on the host name and the current date and time:
# cd /var/log/pcp/pmlogger/# tar -czf $(hostname).$(date +%F-%Hh%M).pcp.tar.gz $(hostname)- Extract this file and analyze the data using PCP tools.
4.3. System services and tools distributed with PCP
The basic package pcp includes the system services and basic tools. You can install additional tools that are provided with the pcp-system-tools, pcp-gui, and pcp-devel packages.
Roles of system services distributed with PCP
pmcd- The Performance Metric Collector Daemon.
pmie- The Performance Metrics Inference Engine.
pmlogger- The performance metrics logger.
pmproxy- The realtime and historical performance metrics proxy, time series query and REST API service.
Tools distributed with base PCP package
pcp- Displays the current status of a Performance Co-Pilot installation.
pcp-check-
Activates, or deactivates core and optional components, such as
pmcd,pmlogger,pmproxy, and PMDAs. pcp-vmstat- Provides a high-level system performance overview every 5 seconds. Displays information about processes, memory, paging, block IO, traps, and CPU activity.
pmconfig- Displays the values of configuration parameters.
pmdiff- Compares average metric values across two archives within a time window to identify potential performance regressions.
pmdumplog- Displays control, metadata, index, and state information from a Performance Co-Pilot archive file.
pmfind- Finds PCP services on the network.
pmie- An inference engine that periodically evaluates a set of arithmetic, logical, and rule expressions. The metrics are collected either from a live system, or from a Performance Co-Pilot archive file.
pmieconf- Displays or sets configurable pmie variables.
pmiectl- Manages non-primary instances of pmie.
pminfo- Displays information about performance metrics. The metrics are collected either from a live system, or from a Performance Co-Pilot archive file.
pmlc- Interactively configures active pmlogger instances.
pmlogcheck- Identifies invalid data in a Performance Co-Pilot archive file.
pmlogconf- Creates and modifies a pmlogger configuration file.
pmlogctl- Manages non-primary instances of pmlogger.
pmloglabel- Verifies, modifies, or repairs the label of a Performance Co-Pilot archive file.
pmlogsummary- Calculates statistical information about performance metrics stored in a Performance Co-Pilot archive file.
pmprobe- Determines the availability of performance metrics.
pmsocks- Provides access to a Performance Co-Pilot hosted through a firewall.
pmstat- Periodically displays a brief summary of system performance.
pmstore- Modifies the values of performance metrics.
pmseries- Fast, scalable time series querying, using the facilities of PCP and a distributed key-value data store such as Valkey.
pmtrace- Provides a command-line interface to the trace PMDA.
pmval- An updating display of the current value of any performance metric.
Tools distributed with the separately installed pcp-system-tools package
pcp-atop- Shows the system-level occupation of the most critical hardware resources from the performance point of view: CPU, memory, disk, and network.
pcp-atopsar- Generates a system-level activity report over a variety of system resource utilization. The report is generated from a raw logfile previously recorded using pmlogger or the -w option of pcp-atop.
pcp-dmcache- Displays information about configured Device Mapper Cache targets. Metrics include device IOPS, utilization, and read/write hit/miss rates for each cache device.
pcp-dstat- Displays metrics of one system at a time. To display metrics of multiple systems, use --host option.
pcp-free- Reports on free and used memory in a system.
pcp-htop-
Lists all processes running on a system along with their command-line arguments. It is similar to the
topcommand, with vertical and horizontal scrolling and mouse interaction. pcp-ipcs- Displays information about the inter-process communication (IPC) facilities that the calling process has read access for.
pcp-mpstat- Reports CPU and interrupt-related statistics.
pcp-numastat- Displays NUMA allocation statistics from the kernel memory allocator.
pcp-pidstat- Displays information about individual tasks or processes running on the system. This includes CPU percentage, memory and stack usage, scheduling, and priority. Reports live data for the local host by default.
pcp-shping- Samples and reports on the shell-ping service metrics exported by the pmdashping Performance Metrics Domain Agent (PMDA).
pcp-ss- Displays socket statistics collected by the pmdasockets PMDA.
pcp-tapestat- Reports I/O statistics for tape devices.
pcp-uptime- Displays the system uptime, currently logged on users, and system load averages for the past 1, 5, and 15 minutes.
pcp-verify- Inspects various aspects of a Performance Co-Pilot collector’s configuration and ensures it is optimized for specific operations.
pcp-iostat- Reports I/O statistics for SCSI devices (by default) or device-mapper devices (with the -x device-mapper option).
pmrep- Reports on selected, easily customizable, performance metrics values.
Tools distributed with the separately installed pcp-gui package
pmchart- Plots performance metrics values available through the facilities of the PCP.
pmdumptext- Outputs the values of performance metrics collected live or from a PCP archive.
Tools distributed with the separately installed pcp-devel package
pmclient- Displays high-level system performance metrics by using the Performance Metrics Application Programming Interface (PMAPI).
pmdbg- Displays available Performance Co-Pilot debug control flags and their values.
pmerr- Displays available Performance Co-Pilot error codes and their corresponding error messages.
pcp-xsos- Gives a fast summary report for a system by using a single sample taken from either a PCP archive or live metrics from the system.
Other tools distributed as a separate packages
pcp-geolocate- Discovers collector system geographical labels.
pcp2openmetrics- A customizable performance metrics exporter tool from PCP to Open Metrics format. You can select any live or archived PCP metric for exporting by using either command line arguments or a configuration file.
4.4. PCP deployment architectures
PCP supports multiple deployment architectures, based on the scale of the PCP deployment, and offers many options to accomplish advanced configurations.
Available scaling deployment configuration variants, determined by sizing factors and configuration options, include the following:
- Localhost
- Each service runs locally on the monitored machine. Starting a service without any configuration changes results in a default standalone deployment on the localhost. This configuration does not support scaling beyond a single node. However, Valkey can also run in a highly available and scalable clustered mode, where data is shared across multiple hosts. You can also deploy a Valkey cluster in the cloud or use a managed Valkey cluster from a cloud provider.
- Decentralized
- The only difference between localhost and decentralized configuration is the centralized Valkey service. In this model, the host runs pmlogger service on each monitored host and retrieves metrics from a local pmcd instance. A local pmproxy service then exports the performance metrics to a central Valkey instance.
Figure 4.1. Decentralized logging
- Centralized logging - pmlogger farm
-
For resource-constrained hosts, use a
pmloggerfarm. This deployment is also called centralized logging. A single logger host runspmloggerprocesses to collect metrics from multiple remote hosts. The centralized logger host runs thepmproxyservice, which discovers the resulting PCP archives and loads the metric data into a Valkey instance.
Figure 4.2. Centralized logging - pmlogger farm
- Federated - multiple pmlogger farms
- For large scale deployments, deploy multiple pmlogger farms in a federated fashion. For example, one pmlogger farm per rack or data center. Each pmlogger farm loads the metrics into a central Valkey instance.
Figure 4.3. Federated - multiple pmlogger farms
By default, the deployment configuration for Valkey is standalone, localhost. However, Valkey can optionally perform in a highly-available and highly scalable clustered fashion, where data is shared across multiple hosts. Another option is to deploy a Valkey cluster in the cloud, or to use a managed Valkey cluster from a cloud provider.
4.5. Factors affecting scaling in PCP logging
The key factors influencing Performance Co-Pilot (PCP) logging are hardware resources, logged metrics, logging intervals, and post-upgrade archive management.
- Remote system size
-
Remote hardware such as CPUs, disks, and network interfaces, determines the volume of data collected by each
pmloggerinstance. - Logged metrics
-
The number and types of logged metrics significantly affect storage requirements. In particular, the
per-process proc.*metrics require a large amount of disk space. For example, a standardpcp-zeroconflogging at 10-second intervals uses 11 MB. Enabling proc metrics increases this to 155 MB. Additionally, the number of CPUs, block devices, and network interfaces impacts storage capacity requirements. - Logging interval
-
The frequency of metric logging determines storage usage. The expected daily PCP archive file sizes for each
pmloggerinstance are recorded in thepmlogger.logfile. PCP archives typically compress at 10:1. Use this ratio to estimate long-term disk space requirements for uncompressed data. - Managing archive updates with
pmlogrewrite -
After PCP upgrades,
pmlogrewriteupdates existing archives if changes are detected in the metric metadata between versions. The time required for this process scales linearly with the number of stored archives.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}