Chapter 7. Logging performance data with pmlogger
With the PCP tool you can log the performance metric values and replay them later. This allows you to perform a retrospective performance analysis.
Using the pmlogger
tool, you can:
- Create the archived logs of selected metrics on the system
- Specify which metrics are recorded on the system and how often
7.1. Modifying the pmlogger configuration file with pmlogconf
When the pmlogger
service is running, PCP logs a default set of metrics on the host.
Use the pmlogconf
utility to check the default configuration. If the pmlogger
configuration file does not exist, pmlogconf
creates it with a default metric values.
Prerequisites
- PCP is installed. For more information, see Installing and enabling PCP.
Procedure
Create or modify the
pmlogger
configuration file:Copy to Clipboard Copied! Toggle word wrap Toggle overflow pmlogconf -r /var/lib/pcp/config/pmlogger/config.default
# pmlogconf -r /var/lib/pcp/config/pmlogger/config.default
-
Follow
pmlogconf
prompts to enable or disable groups of related performance metrics and to control the logging interval for each enabled group.
Additional resources
-
pmlogconf(1)
andpmlogger(1)
man pages on your system - System services and tools distributed with PCP
7.2. Editing the pmlogger configuration file manually
To create a tailored logging configuration with specific metrics and given intervals, edit the pmlogger
configuration file manually. The default pmlogger
configuration file is /var/lib/pcp/config/pmlogger/config.default
. The configuration file specifies which metrics are logged by the primary logging instance.
In manual configuration, you can:
- Record metrics which are not listed in the automatic configuration.
- Choose custom logging frequencies.
- Add PMDA with the application metrics.
Prerequisites
- PCP is installed. For more information, see Installing and enabling PCP.
Procedure
Open and edit the
/var/lib/pcp/config/pmlogger/config.default
file to add specific metrics:Copy to Clipboard Copied! Toggle word wrap Toggle overflow It is safe to make additions from here on ...
# It is safe to make additions from here on ... # log mandatory on every 5 seconds { xfs.write xfs.write_bytes xfs.read xfs.read_bytes } log mandatory on every 10 seconds { xfs.allocs xfs.block_map xfs.transactions xfs.log } [access] disallow * : all; allow localhost : enquire;
Additional resources
-
pmlogger(1)
man page on your system - System services and tools distributed with PCP
7.3. Enabling the pmlogger service
The pmlogger
service must be started and enabled to log the metric values on the local machine.
This procedure describes how to enable the pmlogger
service.
Prerequisites
- PCP is installed. For more information, see Installing and enabling PCP.
Procedure
Start and enable the
pmlogger
service:Copy to Clipboard Copied! Toggle word wrap Toggle overflow systemctl start pmlogger systemctl enable pmlogger
# systemctl start pmlogger # systemctl enable pmlogger
Verification
Verify if the
pmlogger
service is enabled:Copy to Clipboard Copied! Toggle word wrap Toggle overflow pcp
# pcp Performance Co-Pilot configuration on workstation: platform: Linux workstation 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64 hardware: 12 cpus, 2 disks, 1 node, 36023MB RAM timezone: CEST-2 services: pmcd pmcd: Version 4.3.0-1, 8 agents, 1 client pmda: root pmcd proc xfs linux mmv kvm jbd2 pmlogger: primary logger: /var/log/pcp/pmlogger/workstation/20190827.15.54
Additional resources
-
pmlogger(1)
man page on your system - System services and tools distributed with PCP
-
/var/lib/pcp/config/pmlogger/config.default
file
7.4. Setting up a client system for metrics collection
This procedure describes how to set up a client system so that a central server can collect metrics from clients running PCP.
Prerequisites
- PCP is installed. For more information, see Installing and enabling PCP.
Procedure
Install the
pcp-system-tools
package:Copy to Clipboard Copied! Toggle word wrap Toggle overflow dnf install pcp-system-tools
# dnf install pcp-system-tools
Configure an IP address for
pmcd
:Copy to Clipboard Copied! Toggle word wrap Toggle overflow echo "-i 192.168.4.62" >>/etc/pcp/pmcd/pmcd.options
# echo "-i 192.168.4.62" >>/etc/pcp/pmcd/pmcd.options
Replace 192.168.4.62 with the IP address, the client should listen on.
By default,
pmcd
is listening on the localhost.Configure the firewall to add the public
zone
permanently:Copy to Clipboard Copied! Toggle word wrap Toggle overflow firewall-cmd --permanent --zone=public --add-port=44321/tcp firewall-cmd --reload
# firewall-cmd --permanent --zone=public --add-port=44321/tcp success # firewall-cmd --reload success
Set an SELinux boolean:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow setsebool -P pcp_bind_all_unreserved_ports on
# setsebool -P pcp_bind_all_unreserved_ports on
Enable the
pmcd
andpmlogger
services:Copy to Clipboard Copied! Toggle word wrap Toggle overflow systemctl enable pmcd pmlogger systemctl restart pmcd pmlogger
# systemctl enable pmcd pmlogger # systemctl restart pmcd pmlogger
Verification
Verify if the
pmcd
is correctly listening on the configured IP address:Copy to Clipboard Copied! Toggle word wrap Toggle overflow ss -tlp | grep 44321
# ss -tlp | grep 44321 LISTEN 0 5 127.0.0.1:44321 0.0.0.0:* users:(("pmcd",pid=151595,fd=6)) LISTEN 0 5 192.168.4.62:44321 0.0.0.0:* users:(("pmcd",pid=151595,fd=0)) LISTEN 0 5 [::1]:44321 [::]:* users:(("pmcd",pid=151595,fd=7))
Additional resources
-
pmlogger(1)
,firewall-cmd(1)
,ss(8)
, andsetsebool(8)
man pages on your system - System services and tools distributed with PCP
-
/var/lib/pcp/config/pmlogger/config.default
file
7.5. Setting up a central server to collect data
This procedure describes how to create a central server to collect metrics from clients running PCP.
Prerequisites
- PCP is installed. For more information, see Installing and enabling PCP.
- Client is configured for metrics collection. For more information, see Setting up a client system for metrics collection.
Procedure
Install the
pcp-system-tools
package:Copy to Clipboard Copied! Toggle word wrap Toggle overflow dnf install pcp-system-tools
# dnf install pcp-system-tools
Create the
/etc/pcp/pmlogger/control.d/remote
file with the following content:Copy to Clipboard Copied! Toggle word wrap Toggle overflow DO NOT REMOVE OR EDIT THE FOLLOWING LINE
# DO NOT REMOVE OR EDIT THE FOLLOWING LINE $version=1.1 192.168.4.13 n n PCP_ARCHIVE_DIR/rhel7u4a -r -T24h10m -c config.rhel7u4a 192.168.4.14 n n PCP_ARCHIVE_DIR/rhel6u10a -r -T24h10m -c config.rhel6u10a 192.168.4.62 n n PCP_ARCHIVE_DIR/rhel8u1a -r -T24h10m -c config.rhel8u1a 192.168.4.69 n n PCP_ARCHIVE_DIR/rhel9u3a -r -T24h10m -c config.rhel9u3a
Replace 192.168.4.13, 192.168.4.14, 192.168.4.62 and 192.168.4.69 with the client IP addresses.
Enable the
pmcd
andpmlogger
services:Copy to Clipboard Copied! Toggle word wrap Toggle overflow systemctl enable pmcd pmlogger systemctl restart pmcd pmlogger
# systemctl enable pmcd pmlogger # systemctl restart pmcd pmlogger
Verification
Ensure that you can access the latest archive file from each directory:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow for i in /var/log/pcp/pmlogger/rhel*/*.0; do pmdumplog -L $i; done
# for i in /var/log/pcp/pmlogger/rhel*/*.0; do pmdumplog -L $i; done Log Label (Log Format Version 2) Performance metrics from host rhel6u10a.local commencing Mon Nov 25 21:55:04.851 2019 ending Mon Nov 25 22:06:04.874 2019 Archive timezone: JST-9 PID for pmlogger: 24002 Log Label (Log Format Version 2) Performance metrics from host rhel7u4a commencing Tue Nov 26 06:49:24.954 2019 ending Tue Nov 26 07:06:24.979 2019 Archive timezone: CET-1 PID for pmlogger: 10941 [..]
The archive files from the
/var/log/pcp/pmlogger/
directory can be used for further analysis and graphing.
Additional resources
-
pmlogger(1)
man page on your system - System services and tools distributed with PCP
-
/var/lib/pcp/config/pmlogger/config.default
file
7.6. Systemd
units and pmlogger
When you deploy the pmlogger
service, either as a single host monitoring itself or a pmlogger
farm with a single host collecting metrics from several remote hosts, there are several associated systemd
service and timer units that are automatically deployed. These services and timers provide routine checks to ensure that your pmlogger
instances are running, restart any missing instances, and perform archive management such as file compression.
The checking and housekeeping services typically deployed by pmlogger
are:
pmlogger_daily.service
-
Runs daily, soon after midnight by default, to aggregate, compress, and rotate one or more sets of PCP archives. Also culls archives older than the limit, 2 weeks by default. Triggered by the
pmlogger_daily.timer
unit, which is required by thepmlogger.service
unit. pmlogger_check
-
Performs half-hourly checks that
pmlogger
instances are running. Restarts any missing instances and performs any required compression tasks. Triggered by thepmlogger_check.timer
unit, which is required by thepmlogger.service
unit. pmlogger_farm_check
-
Checks the status of all configured
pmlogger
instances. Restarts any missing instances. Migrates all non–primary instances to thepmlogger_farm
service. Triggered by thepmlogger_farm_check.timer
, which is required by thepmlogger_farm.service
unit that is itself required by thepmlogger.service
unit.
These services are managed through a series of positive dependencies, meaning that they are all enabled upon activating the primary pmlogger
instance. Note that while pmlogger_daily.service
is disabled by default, pmlogger_daily.timer
being active via the dependency with pmlogger.service
will trigger pmlogger_daily.service
to run.
pmlogger_daily
is also integrated with pmlogrewrite
for automatically rewriting archives before merging. This helps to ensure metadata consistency amid changing production environments and PMDAs. For example, if pmcd
on one monitored host is updated during the logging interval, the semantics for some metrics on the host might be updated, thus making the new archives incompatible with the previously recorded archives from that host. For more information see the pmlogrewrite(1)
man page.
Managing systemd
services triggered by pmlogger
You can create an automated custom archive management system for data collected by your pmlogger
instances. This is done using control files. These control files are:
For the primary
pmlogger
instance:-
etc/pcp/pmlogger/control
-
/etc/pcp/pmlogger/control.d/local
-
For the remote hosts:
/etc/pcp/pmlogger/control.d/remote
Replace remote with your desired file name.
- NOTE
-
The primary
pmlogger
instance must be running on the same host as thepmcd
it connects to. You do not need to have a primary instance and you might not need it in your configuration if one central host is collecting data on severalpmlogger
instances connected topmcd
instances running on remote host
The file should contain one line for each host to be logged. The default format of the primary logger instance that is automatically created looks similar to:
=== LOGGER CONTROL SPECIFICATIONS === local primary logger
# === LOGGER CONTROL SPECIFICATIONS ===
#
#Host P? S? directory args
# local primary logger
LOCALHOSTNAME y n PCP_ARCHIVE_DIR/LOCALHOSTNAME -r -T24h10m -c config.default -v 100Mb
The fields are:
Host
- The name of the host to be logged
P?
-
Stands for “Primary?” This field indicates if the host is the primary logger instance,
y
, or not,n
. There can only be one primary logger across all the files in your configuration and it must be running on the same host as thepmcd
it connects to. S?
-
Stands for “Socks?” This field indicates if this logger instance needs to use the
SOCKS
protocol to connect topmcd
through a firewall,y
, or not,n
. directory
- All archives associated with this line are created in this directory.
args
Arguments passed to
pmlogger
.The default values for the
args
field are:-r
- Report the archive sizes and growth rate.
T24h10m
-
Specifies when to end logging for each day. This is typically the time when
pmlogger_daily.service
runs. The default value of24h10m
indicates that logging should end 24 hours and 10 minutes after it begins, at the latest. -c config.default
- Specifies which configuration file to use. This essentially defines what metrics to record.
-v 100Mb
-
Specifies the size at which point one data volume is filled and another is created. After it switches to the new archive, the previously recorded one will be compressed by either
pmlogger_daily
orpmlogger_check
.
Additional resources
-
pmlogger(1)
andpmlogrewrite(1)
man pages on your system -
pmlogger_daily(1)
,pmlogger_check(1)
, andpmlogger.control(5)
man pages on your system
7.7. Replaying the PCP log archives with pmrep
After recording the metric data, you can replay the PCP log archives. To export the logs to text files and import them into spreadsheets, use PCP utilities such as pcp2csv
, pcp2xml
, pmrep
or pmlogsummary
.
Using the pmrep
tool, you can:
- View the log files
- Parse the selected PCP log archive and export the values into an ASCII table
- Extract the entire archive log or only select metric values from the log by specifying individual metrics on the command line
Prerequisites
- PCP is installed. For more information, see Installing and enabling PCP.
-
The
pmlogger
service is enabled. For more information, see Enabling the pmlogger service. Install the
pcp-gui
package:Copy to Clipboard Copied! Toggle word wrap Toggle overflow dnf install pcp-gui
# dnf install pcp-gui
Procedure
Display the data on the metric:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow pmrep --start @3:00am --archive 20211128 --interval 5seconds --samples 10 --output csv disk.dev.write
$ pmrep --start @3:00am --archive 20211128 --interval 5seconds --samples 10 --output csv disk.dev.write Time,"disk.dev.write-sda","disk.dev.write-sdb" 2021-11-28 03:00:00,, 2021-11-28 03:00:05,4.000,5.200 2021-11-28 03:00:10,1.600,7.600 2021-11-28 03:00:15,0.800,7.100 2021-11-28 03:00:20,16.600,8.400 2021-11-28 03:00:25,21.400,7.200 2021-11-28 03:00:30,21.200,6.800 2021-11-28 03:00:35,21.000,27.600 2021-11-28 03:00:40,12.400,33.800 2021-11-28 03:00:45,9.800,20.600
The mentioned example displays the data on the
disk.dev.write
metric collected in an archive at a 5 second interval in comma-separated-value format.NoteReplace
20211128
in this example with a filename containing thepmlogger
archive you want to display data for.
Additional resources
-
pmlogger(1)
,pmrep(1)
, andpmlogsummary(1)
man pages on your system - System services and tools distributed with PCP
7.8. Enabling PCP version 3 archives
Performance Co-Pilot (PCP) archives store historical values of PCP metrics recorded from a single host and support retrospective performance analysis. PCP archives contain all the important metric data and metadata needed for offline or offsite analysis. These archives can be read by most PCP client tools or dumped raw by the pmdumplog
tool.
From PCP 6.0, version 3 archives are supported in addition to version 2 archives. Version 2 archives remain the default and will continue to receive long-term support for backwards compatibility purposes in addition to version 3 archives receiving long-term support from RHEL 9.2 and on.
Using PCP version 3 archives offers the following benefits over version 2:
- Support for instance domain change-deltas
- Y2038-safe timestamps
- Nanosecond-precision timestamps
- Arbitrary timezones support
- 64-bit file offsets used for individual volumes larger than 2GB
Prerequisites
- PCP is installed. For more information, see Installing and enabling PCP.
Procedure
Open the
/etc/pcp.conf
file in a text editor of your choice and set the PCP archive version:Copy to Clipboard Copied! Toggle word wrap Toggle overflow PCP_ARCHIVE_VERSION=3
PCP_ARCHIVE_VERSION=3
Restart the
pmlogger
service to apply your configuration changes:Copy to Clipboard Copied! Toggle word wrap Toggle overflow systemctl restart pmlogger.service
# systemctl restart pmlogger.service
- Create a new PCP archive log using your new configuration. For more information, see Logging performance data with pmlogger.
Verification
Verify the version of the archive created with your new configuration:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow pmloglabel -l /var/log/pcp/pmlogger/20230208
# pmloglabel -l /var/log/pcp/pmlogger/20230208 Log Label (Log Format Version 3) Performance metrics from host host1 commencing Wed Feb 08 00:11:09.396 2023 ending Thu Feb 07 00:13:54.347 2023
Additional resources
-
logarchive(5)
andpmlogger(1)
man pages on your system - Logging performance data with pmlogger