Chapter 7. Monitoring and logging

Pull the Docker image for the Ansible on Clouds operational container which aligns with the version of your foundation deployment.

For example, if your foundation deployment version is 2.3.20230221-00, you must pull the operational image with tag 2.3.20230221.

Use the following commands:

$ export IMAGE=registry.redhat.io/ansible-on-clouds/ansible-on-clouds-ops-rhel8:2.3.20230221
$ docker pull $IMAGE --platform=linux/amd64

For EMEA regions (Europe, Middle East, Africa) run the following command instead:

$ export IMAGE=registry.redhat.io/ansible-on-clouds/ansible-on-clouds-ops-emea-rhel8:2.3.20230221
$ docker pull $IMAGE --platform=linux/amd64

The following commands generate the required data file. These commands create a directory, and an empty data template that, when populated, is used to generate the playbook.

If you do not require a parameter, remove that parameter from the configuration file.

To generate the playbook, run the command generator to generate the CLI command.

docker run --rm -v $(pwd)/command_generator_data:/data $IMAGE command_generator gcp_setup_logging_monitoring \
--data-file /data/logging-monitoring.yml

Provides the following command:

docker run --rm --env PLATFORM=GCP -v </path/to/gcp/service-account.json>:/home/runner/.gcp/credentials:ro \
--env ANSIBLE_CONFIG=../gcp-ansible.cfg --env DEPLOYMENT_NAME=<deployment_name> --env GENERATE_INVENTORY=true  \
$IMAGE redhat.ansible_on_clouds.gcp_setup_logging_monitoring -e 'gcp_deployment_name=<deployment_name> \
gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials  monitoring_enabled=<monitoring_enabled> \
logging_enabled=<logging_enabled> default_collector_interval=<interval>'

Run the supplied command to run the playbook.

$ docker run --rm --env PLATFORM=GCP -v /path/to/credentials:/home/runner/.gcp/credentials:ro \
--env ANSIBLE_CONFIG=../gcp-ansible.cfg --env DEPLOYMENT_NAME=mu-deployment \
--env GENERATE_INVENTORY=true  $IMAGE redhat.ansible_on_clouds.gcp_setup_logging_monitoring \
-e 'gcp_deployment_name=mu-deployment \
gcp_service_account_credentials_json_path=/home/runner/.gcp/credentials  components=["hubs","controllers"]\
monitoring_enabled=True logging_enabled=True default_collector_interval=60s'

The process may take some time, and provides output similar to the following:

TASK [redhat.ansible_on_clouds.setup_logging_monitoring : Update runtime variable logging_enabled] ***
changed: [<user_name> -> localhost]

TASK [redhat.ansible_on_clouds.setup_logging_monitoring : Update runtime variable monitoring_enabled] ***
changed: [<user_name> -> localhost]

PLAY RECAP *********************************************************************
<user_name>     : ok=20   changed=6    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

The file /aap/bootstrap/config_file_templates/<controller|hub>/monitoring.yaml on automation controller or automation hub contains the configuration for collecting and sending Ansible and podman metrics to GCP.

The default configuration for automation controller looks like this:

# This value will be set at deployment time.
# Set to zero if monitoringEnabled is false otherwise 59s
# The collection interval for each collector will be the minimum
# between the defaultCollectorInterval and all send Interval
# of a given collector
# NB: The awx exported should not run on controllers as
# it duplicates the number of records sent to GCP Monitoring

defaultCollectorInterval: $DEFAULT_COLLECTOR_INTERVAL
collectors:
- name: podman
 endpoint: http://localhost:9882/podman/metrics
 enabled: true

# list of metrics to exclude
# excludedMetrics:
# - podman_container_created_seconds

 metrics:
 - name: podman_container_exit_code
   # interval on which the metric must be pushed to gcp
   sendInterval: 59s

The default configuration for automation hub looks like:

# This value will be set at deployment time.
# Set to zero if monitoringEnabled is false otherwise 59s
# The collection interval for each collector will be the minimum
# between the defaultCollectorInterval and all sendInterval
# of a given collector
# NB: The awx exporter should not run on controllers as
# it duplicates the number of records sent to GCP Monitoring

defaultCollectorInterval: 59s
collectors:
- name: awx
 userName: admin
 endpoint: http://<Controller_LB_IP>/api/v2/metrics/
 enabled: true
 metrics:
 - name: awx_inventories_total
   # interval on which the metric must be pushed to gcp
   sendInterval: 59s
- name: podman
 endpoint: http://localhost:9882/podman/metrics
 enabled: true

# list of metrics to exclude
#  excludedMetrics:
#  - podman_container_created_seconds

 metrics:
 - name: podman_container_exit_code
   # interval on which the metric must be pushed to gcp
   sendInterval: 59s

where

collectors is a configuration array with one item per collector, that is, awx and podman.

The awx collector requires authentication and so userName must be set to admin. The password is retrieved from the secret-manager.

The endpoint should not be changed.

defaultCollectorInterval specifies the default interval at which the exporter collects the information from the metric end-point and sends it to Google Cloud Platform Monitoring.

Setting this value to 0 or omitting this attribute disables all collectors.

Each collector can be enabled or disabled separately by setting enabled to true or false.

A collector returns all available metrics grouped by families, but you can exclude the families that should not be sent to Google Cloud Platform Monitoring by adding their name in the array excludedMetrics.

For all other family metrics, you can specify the interval at which you want to collect and send them to the Google Cloud Platform Monitoring. The collector interval is the minimum between all family metrics interval and the defaultCollectorInterval. This to ensure that a collection is made for each set of metrics sent to Google Cloud Platform Monitoring.

The configuration file details can be found here.

The configuration file is located in /etc/google-cloud-ops-agent/config.yml.

This is a symbolic link to the shared disk /aap/bootstrap/config_file_templates/controller/gcp-ops-agent-config.yml or /aap/bootstrap/config_file_templates/hub/gcp-ops-agent-config.yml depending on the component type.

The configuration file contains a number of receivers specifying what should be collected by the ops agent.

Your selection of Connect Logging and Connect Metrics during deployment determines which pipelines are included in the file and therefore which logs and metrics are collected and sent to GCP.

If you need to add more pipelines post-deployment, you can insert them in /aap/bootstrap/config_file_templates/hub|controller/gcp-ops-agent-config.yml.

A crontab job restarts the agent if gcp-ops-agent-config.yml changed in the last 10 minutes. The agent rereads its configuration after a restart.