Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 7. Logging performance data with pmlogger

With the PCP tool you can log the performance metric values and replay them later. This allows you to perform a retrospective performance analysis.

Using the pmlogger tool, you can:

  • Create the archived logs of selected metrics on the system
  • Specify which metrics are recorded on the system and how often

7.1. Modifying the pmlogger configuration file with pmlogconf

When the pmlogger service is running, PCP logs a default set of metrics on the host.

Use the pmlogconf utility to check the default configuration. If the pmlogger configuration file does not exist, pmlogconf creates it with a default metric values.

Prerequisites

Procedure

  1. Create or modify the pmlogger configuration file:

    Copy to Clipboard Toggle word wrap 
    # pmlogconf -r /var/lib/pcp/config/pmlogger/config.default
  2. Follow pmlogconf prompts to enable or disable groups of related performance metrics and to control the logging interval for each enabled group.

Additional resources

7.2. Editing the pmlogger configuration file manually

To create a tailored logging configuration with specific metrics and given intervals, edit the pmlogger configuration file manually. The default pmlogger configuration file is /var/lib/pcp/config/pmlogger/config.default. The configuration file specifies which metrics are logged by the primary logging instance.

In manual configuration, you can:

  • Record metrics which are not listed in the automatic configuration.
  • Choose custom logging frequencies.
  • Add PMDA with the application metrics.

Prerequisites

Procedure

  • Open and edit the /var/lib/pcp/config/pmlogger/config.default file to add specific metrics:

    Copy to Clipboard Toggle word wrap 
    # It is safe to make additions from here on ...
#

log mandatory on every 5 seconds {
    xfs.write
    xfs.write_bytes
    xfs.read
    xfs.read_bytes
}

log mandatory on every 10 seconds {
    xfs.allocs
    xfs.block_map
    xfs.transactions
    xfs.log

}

[access]
disallow * : all;
allow localhost : enquire;

Additional resources

7.3. Enabling the pmlogger service

The pmlogger service must be started and enabled to log the metric values on the local machine.

This procedure describes how to enable the pmlogger service.

Prerequisites

Procedure

  • Start and enable the pmlogger service:

    Copy to Clipboard Toggle word wrap 
    # systemctl start pmlogger

# systemctl enable pmlogger

Verification

  • Verify if the pmlogger service is enabled:

    Copy to Clipboard Toggle word wrap 
    # pcp

Performance Co-Pilot configuration on workstation:

platform: Linux workstation 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64
hardware: 12 cpus, 2 disks, 1 node, 36023MB RAM
timezone: CEST-2
services: pmcd
pmcd: Version 4.3.0-1, 8 agents, 1 client
pmda: root pmcd proc xfs linux mmv kvm jbd2
pmlogger: primary logger: /var/log/pcp/pmlogger/workstation/20190827.15.54

Additional resources

7.4. Setting up a client system for metrics collection

This procedure describes how to set up a client system so that a central server can collect metrics from clients running PCP.

Prerequisites

Procedure

  1. Install the pcp-system-tools package:

    Copy to Clipboard Toggle word wrap 
    # dnf install pcp-system-tools

  2. Configure an IP address for pmcd:

    Copy to Clipboard Toggle word wrap 
    # echo "-i 192.168.4.62" >>/etc/pcp/pmcd/pmcd.options

    Replace 192.168.4.62 with the IP address, the client should listen on.

    By default, pmcd is listening on the localhost.

  3. Configure the firewall to add the public zone permanently:

    Copy to Clipboard Toggle word wrap 
    # firewall-cmd --permanent --zone=public --add-port=44321/tcp
success

# firewall-cmd --reload
success

  4. Set an SELinux boolean:

    Copy to Clipboard Toggle word wrap 
    # setsebool -P pcp_bind_all_unreserved_ports on

  5. Enable the pmcd and pmlogger services:

    Copy to Clipboard Toggle word wrap 
    # systemctl enable pmcd pmlogger
# systemctl restart pmcd pmlogger

Verification

  • Verify if the pmcd is correctly listening on the configured IP address:

    Copy to Clipboard Toggle word wrap 
    # ss -tlp | grep 44321
LISTEN   0   5     127.0.0.1:44321   0.0.0.0:*   users:(("pmcd",pid=151595,fd=6))
LISTEN   0   5  192.168.4.62:44321   0.0.0.0:*   users:(("pmcd",pid=151595,fd=0))
LISTEN   0   5         [::1]:44321      [::]:*   users:(("pmcd",pid=151595,fd=7))

Additional resources

7.5. Setting up a central server to collect data

This procedure describes how to create a central server to collect metrics from clients running PCP.

Prerequisites

Procedure

  1. Install the pcp-system-tools package:

    Copy to Clipboard Toggle word wrap 
    # dnf install pcp-system-tools

  2. Create the /etc/pcp/pmlogger/control.d/remote file with the following content:

    Copy to Clipboard Toggle word wrap 
    # DO NOT REMOVE OR EDIT THE FOLLOWING LINE
$version=1.1

192.168.4.13 n n PCP_ARCHIVE_DIR/rhel7u4a -r -T24h10m -c config.rhel7u4a
192.168.4.14 n n PCP_ARCHIVE_DIR/rhel6u10a -r -T24h10m -c config.rhel6u10a
192.168.4.62 n n PCP_ARCHIVE_DIR/rhel8u1a -r -T24h10m -c config.rhel8u1a
192.168.4.69 n n PCP_ARCHIVE_DIR/rhel9u3a -r -T24h10m -c config.rhel9u3a

    Replace 192.168.4.13, 192.168.4.14, 192.168.4.62 and 192.168.4.69 with the client IP addresses.

  3. Enable the pmcd and pmlogger services:

    Copy to Clipboard Toggle word wrap 
    # systemctl enable pmcd pmlogger
# systemctl restart pmcd pmlogger

Verification

  • Ensure that you can access the latest archive file from each directory:

    Copy to Clipboard Toggle word wrap 
    # for i in /var/log/pcp/pmlogger/rhel*/*.0; do pmdumplog -L $i; done
Log Label (Log Format Version 2)
Performance metrics from host rhel6u10a.local
  commencing Mon Nov 25 21:55:04.851 2019
  ending     Mon Nov 25 22:06:04.874 2019
Archive timezone: JST-9
PID for pmlogger: 24002
Log Label (Log Format Version 2)
Performance metrics from host rhel7u4a
  commencing Tue Nov 26 06:49:24.954 2019
  ending     Tue Nov 26 07:06:24.979 2019
Archive timezone: CET-1
PID for pmlogger: 10941
[..]

    The archive files from the /var/log/pcp/pmlogger/ directory can be used for further analysis and graphing.

Additional resources

7.6. Systemd units and pmlogger

When you deploy the pmlogger service, either as a single host monitoring itself or a pmlogger farm with a single host collecting metrics from several remote hosts, there are several associated systemd service and timer units that are automatically deployed. These services and timers provide routine checks to ensure that your pmlogger instances are running, restart any missing instances, and perform archive management such as file compression.

The checking and housekeeping services typically deployed by pmlogger are:

pmlogger_daily.service
Runs daily, soon after midnight by default, to aggregate, compress, and rotate one or more sets of PCP archives. Also culls archives older than the limit, 2 weeks by default. Triggered by the pmlogger_daily.timer unit, which is required by the pmlogger.service unit.
pmlogger_check
Performs half-hourly checks that pmlogger instances are running. Restarts any missing instances and performs any required compression tasks. Triggered by the pmlogger_check.timer unit, which is required by the pmlogger.service unit.
pmlogger_farm_check
Checks the status of all configured pmlogger instances. Restarts any missing instances. Migrates all non–primary instances to the pmlogger_farm service. Triggered by the pmlogger_farm_check.timer, which is required by the pmlogger_farm.service unit that is itself required by the pmlogger.service unit.

These services are managed through a series of positive dependencies, meaning that they are all enabled upon activating the primary pmlogger instance. Note that while pmlogger_daily.service is disabled by default, pmlogger_daily.timer being active via the dependency with pmlogger.service will trigger pmlogger_daily.service to run.

pmlogger_daily is also integrated with pmlogrewrite for automatically rewriting archives before merging. This helps to ensure metadata consistency amid changing production environments and PMDAs. For example, if pmcd on one monitored host is updated during the logging interval, the semantics for some metrics on the host might be updated, thus making the new archives incompatible with the previously recorded archives from that host. For more information see the pmlogrewrite(1) man page.

Managing systemd services triggered by pmlogger

You can create an automated custom archive management system for data collected by your pmlogger instances. This is done using control files. These control files are:

  • For the primary pmlogger instance:

    • etc/pcp/pmlogger/control
    • /etc/pcp/pmlogger/control.d/local

  • For the remote hosts:

    • /etc/pcp/pmlogger/control.d/remote

      Replace remote with your desired file name.

      NOTE
      The primary pmlogger instance must be running on the same host as the pmcd it connects to. You do not need to have a primary instance and you might not need it in your configuration if one central host is collecting data on several pmlogger instances connected to pmcd instances running on remote host

The file should contain one line for each host to be logged. The default format of the primary logger instance that is automatically created looks similar to:

Copy to Clipboard Toggle word wrap 
# === LOGGER CONTROL SPECIFICATIONS ===
#
#Host   	 P?  S?    directory   		 args

# local primary logger
LOCALHOSTNAME    y   n    PCP_ARCHIVE_DIR/LOCALHOSTNAME    -r -T24h10m -c config.default -v 100Mb

The fields are:

Host
The name of the host to be logged
P?
Stands for “Primary?” This field indicates if the host is the primary logger instance, y, or not, n. There can only be one primary logger across all the files in your configuration and it must be running on the same host as the pmcd it connects to.
S?
Stands for “Socks?” This field indicates if this logger instance needs to use the SOCKS protocol to connect to pmcd through a firewall, y, or not, n.
directory
All archives associated with this line are created in this directory.
args

Arguments passed to pmlogger.

The default values for the args field are:

-r
Report the archive sizes and growth rate.
T24h10m
Specifies when to end logging for each day. This is typically the time when pmlogger_daily.service runs. The default value of 24h10m indicates that logging should end 24 hours and 10 minutes after it begins, at the latest.
-c config.default
Specifies which configuration file to use. This essentially defines what metrics to record.
-v 100Mb
Specifies the size at which point one data volume is filled and another is created. After it switches to the new archive, the previously recorded one will be compressed by either pmlogger_daily or pmlogger_check.

Additional resources

  • pmlogger(1) and pmlogrewrite(1) man pages on your system
  • pmlogger_daily(1), pmlogger_check(1), and pmlogger.control(5) man pages on your system

7.7. Replaying the PCP log archives with pmrep

After recording the metric data, you can replay the PCP log archives. To export the logs to text files and import them into spreadsheets, use PCP utilities such as pcp2csv, pcp2xml, pmrep or pmlogsummary.

Using the pmrep tool, you can:

  • View the log files
  • Parse the selected PCP log archive and export the values into an ASCII table
  • Extract the entire archive log or only select metric values from the log by specifying individual metrics on the command line

Prerequisites

  • PCP is installed. For more information, see Installing and enabling PCP.
  • The pmlogger service is enabled. For more information, see Enabling the pmlogger service.

  • Install the pcp-gui package:

    Copy to Clipboard Toggle word wrap 
    # dnf install pcp-gui

Procedure

  • Display the data on the metric:

    Copy to Clipboard Toggle word wrap 
    $ pmrep --start @3:00am --archive 20211128 --interval 5seconds --samples 10 --output csv disk.dev.write
Time,"disk.dev.write-sda","disk.dev.write-sdb"
2021-11-28 03:00:00,,
2021-11-28 03:00:05,4.000,5.200
2021-11-28 03:00:10,1.600,7.600
2021-11-28 03:00:15,0.800,7.100
2021-11-28 03:00:20,16.600,8.400
2021-11-28 03:00:25,21.400,7.200
2021-11-28 03:00:30,21.200,6.800
2021-11-28 03:00:35,21.000,27.600
2021-11-28 03:00:40,12.400,33.800
2021-11-28 03:00:45,9.800,20.600

    The mentioned example displays the data on the disk.dev.write metric collected in an archive at a 5 second interval in comma-separated-value format.

    Note

    Replace 20211128 in this example with a filename containing the pmlogger archive you want to display data for.

Additional resources

7.8. Enabling PCP version 3 archives

Performance Co-Pilot (PCP) archives store historical values of PCP metrics recorded from a single host and support retrospective performance analysis. PCP archives contain all the important metric data and metadata needed for offline or offsite analysis. These archives can be read by most PCP client tools or dumped raw by the pmdumplog tool.

From PCP 6.0, version 3 archives are supported in addition to version 2 archives. Version 2 archives remain the default and will continue to receive long-term support for backwards compatibility purposes in addition to version 3 archives receiving long-term support from RHEL 9.2 and on.

Using PCP version 3 archives offers the following benefits over version 2:

  • Support for instance domain change-deltas
  • Y2038-safe timestamps
  • Nanosecond-precision timestamps
  • Arbitrary timezones support
  • 64-bit file offsets used for individual volumes larger than 2GB

Prerequisites

Procedure

  1. Open the /etc/pcp.conf file in a text editor of your choice and set the PCP archive version:

    Copy to Clipboard Toggle word wrap 
    PCP_ARCHIVE_VERSION=3

  2. Restart the pmlogger service to apply your configuration changes:

    Copy to Clipboard Toggle word wrap 
    # systemctl restart pmlogger.service
  3. Create a new PCP archive log using your new configuration. For more information, see Logging performance data with pmlogger.

Verification

  • Verify the version of the archive created with your new configuration:

    Copy to Clipboard Toggle word wrap 
    # pmloglabel -l /var/log/pcp/pmlogger/20230208
Log Label (Log Format Version 3)
Performance metrics from host host1
        commencing Wed Feb   08 00:11:09.396 2023
        ending           Thu  Feb   07 00:13:54.347 2023

Additional resources

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat, Inc.