5.2. XFS File System Performance Analysis with Performance Co-Pilot
This section describes PCP XFS performance metrics and how to use them. Once started, the Performance Metric Collector Daemon (PMCD) begins collecting performance data from the installed Performance Metric Domain Agents (PMDAs). PMDAs can be individually loaded or unloaded on the system and are controlled by the PMCD on the same host. The XFS PMDA, which is part of the default PCP installation, is used to gather performance metric data of XFS file systems in PCP.
For a list of system services and tools that are distributed with PCP, see Table A.1, “System Services Distributed with Performance Co-Pilot in Red Hat Enterprise Linux 7” and Table A.2, “Tools Distributed with Performance Co-Pilot in Red Hat Enterprise Linux 7”.
5.2.1. Installing XFS PMDA to Gather XFS Data with PCP
The XFS PMDA ships as part of the pcp package and is enabled by default on installation. To install PCP, enter:
#yum install pcp
To enable and start the PMDA service on the host machine after the pcp and pcp-gui packages are installed, use the following commands:
#systemctl enable pmcd.service
#systemctl start pmcd.service
To query the PCP environment to verify that the PMCD process is running on the host and that the XFS PMDA is listed as enabled in the configuration, enter:
#pcpPerformance Co-Pilot configuration on workstation: platform: Linux workstation 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64 hardware: 2 cpus, 2 disks, 1 node, 2048MB RAM timezone: BST-1 services pmcd pmcd: Version 3.10.6-1, 7 agents pmda: root pmcd proc xfs linux mmv jbd2
Installing XFS PMDA Manually
If the XFS PMDA is not listed in PCP configuration readout, install the PMDA agent manually. The PMDA installation script prompts you to specify the PMDA role: collector, monitor, or both.
The default option is
- The
collectorrole allows the collection of performance metrics on the current system
- The
monitorrole allows the system to monitor local systems, remote systems, or both.
both collector and monitor, which allows the XFS PMDA to operate correctly in most scenarios.
To install XFS PMDA manually, change to the xfs directory:
In the#cd /var/lib/pcp/pmdas/xfs/
xfs directory, enter:
xfs]#./InstallYou will need to choose an appropriate configuration for install of the “xfs” Performance Metrics Domain Agent (PMDA). collector collect performance statistics on this system monitor allow this system to monitor local and/or remote systems both collector and monitor configuration for this system Please enter c(ollector) or m(onitor) or (both) [b] Updating the Performance Metrics Name Space (PMNS) ... Terminate PMDA if already installed ... Updating the PMCD control file, and notifying PMCD ... Waiting for pmcd to terminate ... Starting pmcd ... Check xfs metrics have appeared ... 149 metrics and 149 values
5.2.2. Configuring and Examining XFS Performance Metrics
Examining Metrics with pminfo
With PCP installed and the XFS PMDA enabled, instructions are available in Section 5.2.1, “Installing XFS PMDA to Gather XFS Data with PCP”, the easiest way to start looking at the performance metrics available for PCP and XFS is to use the
pminfo tool, which displays information about available performance metrics. The command displays a list of all available metrics provided by the XFS PMDA.
To display a list of all available metrics provided by the XFS PMDA:
#pminfo xfs
Use the following options to display information on selected metrics:
-t metric- Displays one-line help information describing the selected metric.
-T metric- Displays more verbose help text describing the selected metric.
-f metric- Displays the current reading of the performance value that corresponds to the metric.
You can use the
-t, -T, and -f options with a group of metrics or an individual metric. Most metric data is provided for each mounted XFS file system on the system at time of probing.
There are different groups of XFS metrics, which are arranged so that each different group is a new leaf node from the root XFS metric, using a dot (
.) as a separator. The leaf node semantics (dots) applies to all PCP metrics. For an overview of the types of metrics that are available in each of the groups, see Table A.3, “PCP Metric Groups for XFS”.
Additionally, the XFS documentation contains a section on monitoring XFS file systems: Chapter 13. XFS Monitoring.
Example 5.1. Using the pminfo Tool to Examine XFS Read and Write Metrics
To display one-line help information describing the
xfs.write_bytes metric:
#pminfo -t xfs.write_bytesxfs.write_bytes [number of bytes written in XFS file system write operations]
To display more verbose help text describing the
xfs.read_bytes metric:
#pminfo -T xfs.read_bytesxfs.read_bytes Help: This is the number of bytes read via read(2) system calls to files in XFS file systems. It can be used in conjunction with the read_calls count to calculate the average size of the read operations to file in XFS file systems.
To obtain the current reading of the performance value that corresponds to the
xfs.read_bytes metric:
#pminfo -f xfs.read_bytesxfs.read_bytes value 4891346238
Configuring Metrics with pmstore
With PCP, you can modify the values of certain metrics, especially if the metric acts as a control variable, for example the
xfs.control.reset metric. To modify a metric value, use the pmstore tool.
Example 5.2. Using pmstore to Reset the xfs.control.reset Metric
This example shows how to use
pmstore with the xfs.control.reset metric to reset the recorded counter values for the XFS PMDA back to zero.
$pminfo -f xfs.writexfs.write value 325262
#pmstore xfs.control.reset 1xfs.control.reset old value=0 new value=1
$pminfo -f xfs.writexfs.write value 0
5.2.3. Examining XFS Metrics Available per File System
Starting with Red Hat Enterprise Linux 7.3, PCP enables XFS PMDA to allow the reporting of certain XFS metrics per each of the mounted XFS file systems. This makes it easier to pinpoint specific mounted file system issues and evaluate performance. For an overview of the types of metrics available per file system in each of the groups, see Table A.4, “PCP Metric Groups for XFS per Device”.
Example 5.3. Obtaining per-Device XFS Metrics with pminfo
The
pminfo command provides per-device XFS metrics that give instance values for each mounted XFS file system.
#pminfo -f -t xfs.perdev.read xfs.perdev.writexfs.perdev.read [number of XFS file system read operations] inst [0 or "loop1"] value 0 inst [0 or "loop2"] value 0 xfs.perdev.write [number of XFS file system write operations] inst [0 or "loop1"] value 86 inst [0 or "loop2"] value 0
5.2.4. Logging Performance Data with pmlogger
PCP allows you to log performance metric values that can be replayed later and used for a retrospective performance analysis. Use the
pmlogger tool to create archived logs of selected metrics on the system.
With pmlogger, you can specify which metrics are recorded on the system and how often. The default pmlogger configuration file is
/var/lib/pcp/config/pmlogger/config.default. The configuration file specifies which metrics are logged by the primary logging instance.
To log metric values on the local machine with
pmlogger, start a primary logging instance:
#systemctl start pmlogger.service
#systemctl enable pmlogger.service
When
pmlogger is enabled and a default configuration file is set, a pmlogger line is included in the PCP configuration:
#pcpPerformance Co-Pilot configuration on workstation: platform: Linux workstation 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan [...] pmlogger: primary logger:/var/log/pcp/pmlogger/workstation/20160820.10.15
Modifying the pmlogger Configuration File with pmlogconf
When the
pmlogger service is running, PCP logs a default set of metrics on the host. You can use the pmlogconf utility to check the default configuration, and enable XFS logging groups as needed. Important XFS groups to enable include the XFS information, XFS data, and log I/O traffic groups.
Follow
pmlogconf prompts to enable or disable groups of related performance metrics, and to control the logging interval for each enabled group. Group selection is made by pressing y (yes) or n (no) in response to the prompt. To create or modify the generic PCP archive logger configuration file with pmlogconf, enter:
#pmlogconf -r /var/lib/pcp/config/pmlogger/config.default
Modifying the pmlogger Configuration File Manually
You can edit the
pmlogger configuration file manually and add specific metrics with given intervals to create a tailored logging configuration.
Example 5.4. The pmlogger Configuration File with XFS Metrics
The following example shows an extract of the
pmlogger config.default file with some specific XFS metrics added.
# It is safe to make additions from here on ...
#
log mandatory on every 5 seconds {
xfs.write
xfs.write_bytes
xfs.read
xfs.read_bytes
}
log mandatory on every 10 seconds {
xfs.allocs
xfs.block_map
xfs.transactions
xfs.log
}
[access]
disallow * : all;
allow localhost : enquire;
Replaying the PCP Log Archives
After recording metric data, you can replay the PCP log archives on the system in the following ways:
- You can export the logs to text files and import them into spreadsheets by using PCP utilities such as
pmdumptext,pmrep, orpmlogsummary. - You can replay the data in the PCP Charts application and use graphs to visualize the retrospective data alongside live data of the system. See Section 5.2.5, “Visual Tracing with PCP Charts”.
You can use the
pmdumptext tool to view the log files. With pmdumptext, you can parse the selected PCP log archive and export the values into an ASCII table. The pmdumptext tool enables you to dump the entire archive log, or only select metric values from the log by specifying individual metrics on the command line.
Example 5.5. Displaying a Specific XFS Metric Log Information
For example, to show data on the
xfs.perdev.log metric collected in an archive at a 5 second interval and display all headers:
$pmdumptext -t 5seconds -H -a 20170605 xfs.perdev.log.writesTime local::xfs.perdev.log.writes["/dev/mapper/fedora-home"] local::xfs.perdev.log.writes["/dev/mapper/fedora-root"] ? 0.000 0.000 none count / second count / second Mon Jun 5 12:28:45 ? ? Mon Jun 5 12:28:50 0.000 0.000 Mon Jun 5 12:28:55 0.200 0.200 Mon Jun 5 12:29:00 6.800 1.000
For more information, see the pmdumptext(1) manual page, which is available from the pcp-doc package.
5.2.5. Visual Tracing with PCP Charts
To be able to use the graphical PCP Charts application, install the pcp-gui package:
#yum install pcp-gui
You can use the PCP Charts application to plot performance metric values into graphs. The PCP Charts application allows multiple charts to be displayed simultaneously. The metrics are sourced from one or more live hosts with alternative options to use metric data from PCP log archives as a source of historical data. To launch PCP Charts from the command line, use the
pmchart command.
After starting PCP Charts, the GUI appears:
The PCP Charts application
The
pmtime server settings are located at the bottom. The and button allows you to control:
- The interval in which PCP polls the metric data
- The date and time for the metrics of historical data
Go to
There are multiple options to take images or record the views created in PCP Charts:
- Click
to save an image of the current view. - Click
to start a recording. Click to stop the recording. After stopping the recording, the recorded metrics are archived to be viewed later.
You can customize the PCP Charts interface to display the data from performance metrics in multiple ways, including:
- line plot
- bar graphs
- utilization graphs
In PCP Charts, the main configuration file, known as the
view, allows the metadata associated with one or more charts to be saved. This metadata describes all chart aspects, including the metrics used and the chart columns. You can create a custom view configuration, save it by clicking view configuration later. For more information about view configuration files and their syntax, see the pmchart(1) manual page.
Example 5.6. Stacking Chart Graph in PCP Charts View Configuration
The example PCP Charts view configuration file describes a stacking chart graph showing the total number of bytes read and written to the given XFS file system
loop1.
#kmchart
version 1
chart title "Filesystem Throughput /loop1" style stacking antialiasing off
plot legend "Read rate" metric xfs.read_bytes instance "loop1"
plot legend "Write rate" metric xfs.write_bytes instance "loop1"
