Monitoring Red Hat Satellite

Procedure

1. Configure PCP to collect data about important Satellite processes.

   By default, PCP collects basic system metrics. This step enables detailed metrics about the following Satellite processes:

   • Java
   • PostgreSQL
   • MongoDB
   • Dynflow
   • Passenger
   • Pulp

   • Qpid

     # cat >/var/lib/pcp/pmdas/proc/hotproc.conf <<EOF
     #pmdahotproc
     Version 1.0
     
     fname == "java" ||
     fname ~ /(qdrouterd|qpidd)/ ||
     (fname == "postgres" && psargs ~ /-D/) ||
     fname == "mongod" ||
     fname ~ /^dynflow/ ||
     psargs ~ /Passenger RackApp/ ||
     fname ~ /^wsgi:pulp/ ||
     psargs ~ /celery (beat|worker)/ ||
     psargs ~ /pulp_streamer/ ||
     psargs ~ /smart-proxy/ ||
     psargs ~ /squid.conf/
     EOF

2. Configure PCP to log the process metrics being collected.

   # mkdir -p /var/lib/pcp/config/pmlogconf/foreman-hotproc
   # cat >/var/lib/pcp/config/pmlogconf/foreman-hotproc/summary << EOF
   #pmlogconf-setup 2.0
   ident   foreman hotproc metrics
   probe   hotproc.control.config != "" ? include : exclude
           hotproc.psinfo.psargs
           hotproc.psinfo.cnswap
           hotproc.psinfo.nswap
           hotproc.psinfo.rss
           hotproc.psinfo.vsize
           hotproc.psinfo.cstime
           hotproc.psinfo.cutime
           hotproc.psinfo.stime
           hotproc.psinfo.utime
           hotproc.io.write_bytes
           hotproc.io.read_bytes
           hotproc.schedstat.cpu_time
           hotproc.fd.count
   EOF

3. Install the process monitoring PMDA.

   # cd /var/lib/pcp/pmdas/proc
   # ./Install

4. Configure PCP to collect metrics from Apache HTTP Server.

   1. Enable the Apache HTTP Server extended status module.

      #cat >/etc/httpd/conf.d/01-status.conf <<EOF
      ExtendedStatus On
      LoadModule status_module modules/mod_status.so
      
      <Location "/server-status">
      PassengerEnabled off
      SetHandler server-status
      Order deny,allow
      Deny from all
      Allow from localhost
      </Location>
      EOF

   2. Enable the Apache HTTP Server PMDA.

      # cd /var/lib/pcp/pmdas/apache
      # ./Install

   3. Prevent the Satellite installer overwriting the extended status module’s configuration file.

      Add the following line to the /etc/foreman-installer/custom-hiera.yaml configuration file.

      apache::purge_configs: false

5. Configure PCP to collect metrics from PostgreSQL.

   1. Change to the /var/lib/pcp/pmdas/postgresql directory.

      # cd /var/lib/pcp/pmdas/postgresql

   2. Run the installer.

      # ./Install

   3. Configure the PCP database interface to permit access to the PostgreSQL database.

      Edit the /etc/pcpdbi.conf configuration file, inserting the following lines:

      $database = "dbi:Pg:dbname=foreman;host=localhost";
      $username = "foreman";
      $password = "6qXfN9m5nii5iEcbz8nuiJBNsyjjdRHA"; 1
      $os_user = "foreman";

      1

      The value for $password is stored in /etc/foreman/database.yml configuration file.

   4. Change the SELinux pcp_pmcd_t domain permission to permit PCP access to the PostgreSQL database.

      # semanage permissive -a pcp_pmcd_t

   5. Verify the PostgreSQL PMDA is able to connect to PostgreSQL.

      Examine the /var/log/pcp/pmcd/postgresql.log file to confirm the connection is established. Without a successful database connection, the PostgreSQL PMDA will remain active, but not be able to provide any metrics.

      [Tue Aug 14 09:21:06] pmdapostgresql(25056) Info: PostgreSQL connection established

      If you find errors in /var/log/pcp/pmcd/postgresql.log, restart the pmcd service.

      # systemctl restart pmcd

6. Enable telemetry functionality in Satellite.

   To enable collection of metrics from Satellite, you must send metrics via the statsd protocol into the pcp-mmvstatsd daemon. The metrics are aggregated and available via the PCP MMV API.

   1. Install the Foreman Telemetry and pcp-mmvstatsd packages.

      # yum install foreman-telemetry pcp-mmvstatsd

   2. Enable and start the pcp-mmvstatsd service.

      # systemctl enable pcp-mmvstatsd
      # systemctl start pcp-mmvstatsd

   3. Enable the Satellite telemetry functionality.

      Add the following lines to /etc/foreman/settings.yaml configuration file:

      :telemetry:
        :prefix: 'fm_rails'
        :statsd:
          :enabled: true
          :host: '127.0.0.1:8125'
          :protocol: 'statsd'
        :prometheus:
          :enabled: false
        :logger:
          :enabled: false
          :level: 'INFO'

7. Schedule daily storage of metrics in archive files:

   # cat >/etc/cron.daily/refresh_mmv <<EOF
   #!/bin/bash
   echo "log mandatory on 1 minute mmv" | /usr/bin/pmlc -P
   EOF
   # chmod +x /etc/cron.daily/refresh_mmv

8. Restart the Apache HTTP Server and PCP to begin data collection:

   # systemctl restart httpd pmcd pmlogger

Procedure

1. Enable the Red Hat Enterprise Linux optional repository:

   # subscription-manager repos --enable rhel-7-server-optional-rpms

2. Install the PCP web API and applications:

   # yum install pcp-webapi pcp-webapp-grafana pcp-webapp-vector

3. Start and enable the PCP web service:

   # systemctl start pmwebd
   # systemctl enable pmwebd

4. Open firewall port to allow access to the PCP web service:

   # firewall-cmd --add-port=44323/tcp
   # firewall-cmd --permanent --add-port=44323/tcp

Procedure

1. Edit the /etc/pcp/pmlogger/control.d/local configuration file.
2. Edit the LOCALHOSTNAME line and append -t XXs, where XX is the desired time interval, measured in seconds.
3. Restart the pmlogger service.

Procedure

1. Edit the /etc/cron.d/pcp-pmlogger file.
2. Find the line containing pmlogger_daily.
3. Change the value for parameter -x to the desired number of days after which data is archived.

4. Add parameter -k, and add a value for the number of days after which data is deleted.

   For example, the parameters -x 4 -k 7 specify that data will be compressed after 4 days, and deleted after 7 days.