Monitoring Red Hat Satellite

Procedure

1. To configure PCP to collect data about Satellite processes, create the /var/lib/pcp/pmdas/proc/hotproc.conf file and include the following content:

   #pmdahotproc
   Version 1.0
   
   # processes with load 0.9 or 1GB RSS memory
   (cpuburn > 0.9 || residentsize > 1000000) &&
   (
     fname == "java" ||
     fname == "qdrouterd" ||
     fname == "qpidd" ||
     (fname == "postgres" && psargs ~ /-D/) ||
     fname == "mongod" ||
     fname == "dynflow_executor" ||
     fname == "dynflow_executor_monitor" ||
     fname ~ /smart_proxy_dynflow_core/ ||
     psargs ~ /Passenger RackApp/ ||
     psargs ~ /celery beat/ ||
     psargs ~ /celery worker/ ||
     psargs ~ /pulp/ ||
     psargs ~ /smart-proxy/ ||
     psargs ~ /squid.conf/
   )

   Copy to Clipboard Toggle word wrap

   By default, PCP collects basic system metrics. This step enables detailed metrics about the following Satellite processes:

   • Java
   • PostgreSQL
   • MongoDB
   • Dynflow
   • Passenger
   • Pulp
   • Qpid

2. Configure PCP to log the process metrics being collected.

   # mkdir -p /var/lib/pcp/config/pmlogconf/foreman-hotproc
   # cat >/var/lib/pcp/config/pmlogconf/foreman-hotproc/summary << EOF
   #pmlogconf-setup 2.0
   ident   foreman hotproc metrics
   probe   hotproc.control.config != "" ? include : exclude
           hotproc.psinfo.psargs
           hotproc.psinfo.cnswap
           hotproc.psinfo.nswap
           hotproc.psinfo.rss
           hotproc.psinfo.vsize
           hotproc.psinfo.cstime
           hotproc.psinfo.cutime
           hotproc.psinfo.stime
           hotproc.psinfo.utime
           hotproc.io.write_bytes
           hotproc.io.read_bytes
           hotproc.schedstat.cpu_time
           hotproc.fd.count
   EOF

   Copy to Clipboard Toggle word wrap

3. Install the process monitoring PMDA.

   # cd /var/lib/pcp/pmdas/proc
   # ./Install

   Copy to Clipboard Toggle word wrap

4. Configure PCP to collect metrics from Apache HTTP Server.

   1. Enable the Apache HTTP Server extended status module.

      #cat >/etc/httpd/conf.d/01-status.conf <<EOF
      ExtendedStatus On
      LoadModule status_module modules/mod_status.so
      
      <Location "/server-status">
      PassengerEnabled off
      SetHandler server-status
      Order deny,allow
      Deny from all
      Allow from localhost
      </Location>
      EOF

      Copy to Clipboard Toggle word wrap

   2. Enable the Apache HTTP Server PMDA.

      # cd /var/lib/pcp/pmdas/apache
      # ./Install

      Copy to Clipboard Toggle word wrap

   3. Prevent the Satellite installer overwriting the extended status module’s configuration file.

      Add the following line to the /etc/foreman-installer/custom-hiera.yaml configuration file.

      apache::purge_configs: false

      Copy to Clipboard Toggle word wrap

5. Configure PCP to collect metrics from PostgreSQL.

   1. Change to the /var/lib/pcp/pmdas/postgresql directory.

      # cd /var/lib/pcp/pmdas/postgresql

      Copy to Clipboard Toggle word wrap

   2. Run the installer.

      # ./Install

      Copy to Clipboard Toggle word wrap

   3. Configure the PCP database interface to permit access to the PostgreSQL database.

      Edit the /etc/pcpdbi.conf configuration file, inserting the following lines:

      $database = "dbi:Pg:dbname=foreman;host=localhost";
      $username = "foreman";
      $password = "6qXfN9m5nii5iEcbz8nuiJBNsyjjdRHA"; 

      1

      
      $os_user = "foreman";

      Copy to Clipboard Toggle word wrap

      1

      The value for $password is stored in /etc/foreman/database.yml configuration file.

   4. Change the SELinux pcp_pmcd_t domain permission to permit PCP access to the PostgreSQL database.

      # semanage permissive -a pcp_pmcd_t

      Copy to Clipboard Toggle word wrap

   5. Verify the PostgreSQL PMDA is able to connect to PostgreSQL.

      Examine the /var/log/pcp/pmcd/postgresql.log file to confirm the connection is established. Without a successful database connection, the PostgreSQL PMDA will remain active, but not be able to provide any metrics.

      [Tue Aug 14 09:21:06] pmdapostgresql(25056) Info: PostgreSQL connection established

      Copy to Clipboard Toggle word wrap

      If you find errors in /var/log/pcp/pmcd/postgresql.log, restart the pmcd service.

      # systemctl restart pmcd

      Copy to Clipboard Toggle word wrap

6. Enable telemetry functionality in Satellite.

   To enable collection of metrics from Satellite, you must send metrics via the statsd protocol into the pcp-mmvstatsd daemon. The metrics are aggregated and available via the PCP MMV API.

   1. Install the Foreman Telemetry and pcp-mmvstatsd packages.

      # yum install foreman-telemetry pcp-mmvstatsd

      Copy to Clipboard Toggle word wrap

   2. Enable and start the pcp-mmvstatsd service.

      # systemctl enable pcp-mmvstatsd
      # systemctl start pcp-mmvstatsd

      Copy to Clipboard Toggle word wrap

   3. Enable the Satellite telemetry functionality.

      Add the following lines to /etc/foreman/settings.yaml configuration file:

      :telemetry:
        :prefix: 'fm_rails'
        :statsd:
          :enabled: true
          :host: '127.0.0.1:8125'
          :protocol: 'statsd'
        :prometheus:
          :enabled: false
        :logger:
          :enabled: false
          :level: 'INFO'

      Copy to Clipboard Toggle word wrap

7. Schedule daily storage of metrics in archive files:

   # cat >/etc/cron.daily/refresh_mmv <<EOF
   #!/bin/bash
   echo "log mandatory on 1 minute mmv" | /usr/bin/pmlc -P
   EOF
   # chmod +x /etc/cron.daily/refresh_mmv

   Copy to Clipboard Toggle word wrap

8. Restart the Apache HTTP Server and PCP to begin data collection:

   # systemctl restart httpd pmcd pmlogger

   Copy to Clipboard Toggle word wrap

Procedure

1. Enable the Red Hat Enterprise Linux optional repository:

   # subscription-manager repos --enable rhel-7-server-optional-rpms

   Copy to Clipboard Toggle word wrap

2. Install the PCP web API and applications:

   # yum install pcp-webapi pcp-webapp-grafana pcp-webapp-vector

   Copy to Clipboard Toggle word wrap

3. Start and enable the PCP web service:

   # systemctl start pmwebd
   # systemctl enable pmwebd

   Copy to Clipboard Toggle word wrap

4. Open firewall port to allow access to the PCP web service:

   # firewall-cmd --add-port=44323/tcp
   # firewall-cmd --permanent --add-port=44323/tcp

   Copy to Clipboard Toggle word wrap

1. Open the following URL: http://satellite.example.com:44323
2. In the PCP web UI, select the web application dashboard that you want to access.

Procedure

1. Edit the /etc/pcp/pmlogger/control.d/local configuration file.
2. Edit the LOCALHOSTNAME line and append -t XXs, where XX is the desired time interval, measured in seconds.
3. Restart the pmlogger service.

Procedure

1. Edit the /etc/cron.d/pcp-pmlogger file.
2. Find the line containing pmlogger_daily.
3. Change the value for parameter -x to the desired number of days after which data is archived.

4. Add parameter -k, and add a value for the number of days after which data is deleted.

   For example, the parameters -x 4 -k 7 specify that data will be compressed after 4 days, and deleted after 7 days.