Chapter 5. PCP metrics
Metrics are stored in a tree-like structure. For example, all network metrics are stored in a node named network. Each metric may be a single value, or a list of values, known as instances. For example, kernel load has three instances, a 1-minute, 5-minute, and 15-minute average.
For every metric entry, PCP stores both its data and metadata. This includes the metrics description, data type, units, and dimensions. For example, the metadata enables PCP to output multiple metrics with different dimensions.
If a metric is a counter, its value can only increase. For example, a count of disk write operations on a specific device only increases. When you query the value of a counter metric, PCP converts this into a rate value by default.
In addition to system metrics, such as CPU, memory, kernel, XFS, disk, and network, the following metrics are configured:
| Metric | Description |
|---|---|
| hotproc.* | Basic metrics of key Satellite processes |
| apache.* | Apache HTTP Server metrics |
| postgresql.* | Basic PostgreSQL statistics |
| openmetrics.foreman.fm_rails_* | Satellite metrics |
5.1. Identifying available metrics
To list all metrics available through PCP, enter the following command:
# pminfo
To list all Satellite metrics and their descriptions, enter the following command:
# foreman-rake telemetry:metrics
To list the archived metrics, enter the following command:
# less /var/log/pcp/pmlogger/satellite.example.com/pmlogger.logThe
pmloggerdaemon archives data as received, according to its configuration. To retrieve the name of the active log file, enter the following command:# pcp | grep logger
The output includes the file name of the active log file, for example:
pmlogger: primary logger: /var/log/pcp/pmlogger/satellite.example.com/20230831.00.10
5.2. Retrieving live metrics
You can use the PCP CLI tools to retrieve live metrics.
Procedures
To print current values of a particular metric, enter the following command:
# pmval -f 1 disk.partitions.write
In this example, metric instances on writes to disk partitions are displayed. PCP converts the number of writes to disk partitions from a counter value to a rate value. The
-f 1argument specifies to abbreviate the values to one decimal place.Example output:
metric: disk.partitions.write host: satellite.example.com semantics: cumulative counter (converting to rate) units: count (converting to count / sec) samples: all vda1 vda2 sr0 0.0 12.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 2.0 0.0To print system performance summary every 2 seconds, enter the following command:
# pmstat -t 2sec
5.3. Retrieving archived metrics
You can use the PCP CLI tools to retrieve metrics from an archive file.
Examples
To list all metrics that were enabled when the archive file was created, enter the following command:
# pminfo --archive archive_fileTo view the host and time period covered by an archive file, enter the following command:
# pmdumplog -l archive_fileTo list disk writes for each partition, over the time period covered by the archive file:
# pmval --archive /var/log/pcp/pmlogger/satellite.example.com/20230831.00.10 \ -f 1 disk.partitions.write
To list disk write operations per partition, with a 2-second interval, over the time period between 14:00 and 14:15:
# pmval --archive /var/log/pcp/pmlogger/satellite.example.com/20230831.00.10 \ -d -t 2sec \ -f 3 disk.partitions.write \ -S @14:00 -T @14:15
To list average values of all performance metrics, including the time and value of the minimum/maximum, over the time period between 14:00 and 14:30, and format the values as a table:
# pmlogsummary /var/log/pcp/pmlogger/satellite.example.com/20230831.00.10 \ -HlfiImM \ -S @14:00 \ -T @14:30 \ disk.partitions.write \ mem.freemem
To display system metrics stored in an archive, starting from 14:00, in an interactive manner similar to the
toptool:# pcp --archive /var/log/pcp/pmlogger/satellite.example.com/20230831.00.10 \ -S @14:00 \ atop
5.4. Retrieving metrics in the web UI
You can view the Grafana dashboard, which displays widgets with the values of metrics. You can add and remove metrics to suit your requirements. You can also select the time span displayed for each widget.
Procedure
- Open the following URL in a browser: http://satellite.example.com:3000
Additional resources
- For more details on using Grafana, see the Grafana Labs website.
