Chapter 5. Using operational features of Service Telemetry Framework
You can use the following operational features to provide additional functionality to the Service Telemetry Framework (STF):
5.1. Dashboards in Service Telemetry Framework
Use the third-party application, Grafana, to visualize system-level metrics that the data collectors collectd and Ceilometer gather for each individual host node.
For more information about configuring data collectors, see Section 4.1, “Deploying Red Hat OpenStack Platform overcloud for Service Telemetry Framework using director”.
You can use dashboards to monitor a cloud:
- Infrastructure dashboard
- Use the infrastructure dashboard to view metrics for a single node at a time. Select a node from the upper left corner of the dashboard.
- Cloud view dashboard
Use the cloud view dashboard to view panels to monitor service resource usage, API stats, and cloud events. You must enable API health monitoring and service monitoring to provide the data for this dashboard. API health monitoring is enabled by default in the STF base configuration. For more information, see Section 4.1.2, “Creating the base configuration for STF”.
- For more information about API health monitoring, see Section 5.8, “Red Hat OpenStack Platform API status and containerized services health”.
- For more information about RHOSP service monitoring, see Section 5.7, “Resource usage of Red Hat OpenStack Platform services”.
- Virtual machine view dashboard
- Use the virtual machine view dashboard to view panels to monitor virtual machine infrastructure usage. Select a cloud and project from the upper left corner of the dashboard. You must enable event storage if you want to enable the event annotations on this dashboard. For more information, see Section 3.2, “Creating a ServiceTelemetry object in Red Hat OpenShift Container Platform”.
- Memcached view dashboard
- Use the memcached view dashboard to view panels to monitor connections, availability, system metrics and cache performance. Select a cloud from the upper left corner of the dashboard.
5.1.1. Configuring Grafana to host the dashboard
Grafana is not included in the default Service Telemetry Framework (STF) deployment, so you must deploy the Grafana Operator from community-operators CatalogSource. If you use the Service Telemetry Operator to deploy Grafana, it results in a Grafana instance and the configuration of the default data sources for the local STF deployment.
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the
service-telemetry
namespace:$ oc project service-telemetry
Subscribe to the Grafana Operator by using the community-operators CatalogSource:
WarningCommunity Operators are Operators which have not been vetted or verified by Red Hat. Community Operators should be used with caution because their stability is unknown. Red Hat provides no support for community Operators.
Learn more about Red Hat’s third party software support policy
$ oc apply -f - <<EOF apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: grafana-operator namespace: service-telemetry spec: channel: v4 installPlanApproval: Automatic name: grafana-operator source: community-operators sourceNamespace: openshift-marketplace EOF
Verify that the Operator launched successfully. In the command output, if the value of the
PHASE
column isSucceeded
, the Operator launched successfully:$ oc get csv --selector operators.coreos.com/grafana-operator.service-telemetry NAME DISPLAY VERSION REPLACES PHASE grafana-operator.v4.10.1 Grafana Operator 4.10.1 grafana-operator.v4.10.0 Succeeded
To launch a Grafana instance, create or modify the
ServiceTelemetry
object. Setgraphing.enabled
andgraphing.grafana.ingressEnabled
totrue
. Optionally, set the value ofgraphing.grafana.baseImage
to the Grafana workload container image that will be deployed:$ oc edit stf default apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry ... spec: ... graphing: enabled: true grafana: ingressEnabled: true baseImage: 'registry.redhat.io/rhel8/grafana:7'
Verify that the Grafana instance deployed:
$ oc get pod -l app=grafana NAME READY STATUS RESTARTS AGE grafana-deployment-7fc7848b56-sbkhv 1/1 Running 0 1m
Verify that the Grafana data sources installed correctly:
$ oc get grafanadatasources NAME AGE default-datasources 20h
Verify that the Grafana route exists:
$ oc get route grafana-route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD grafana-route grafana-route-service-telemetry.apps.infra.watch grafana-service 3000 edge None
5.1.2. Overriding the default Grafana container image
The dashboards in Service Telemetry Framework (STF) require features that are available only in Grafana version 8.1.0 and later. By default, the Service Telemetry Operator installs a compatible version. You can override the base Grafana image by specifying the image path to an image registry with graphing.grafana.baseImage
.
Procedure
Ensure that you have the correct version of Grafana:
$ oc get pod -l "app=grafana" -ojsonpath='{.items[0].spec.containers[0].image}' docker.io/grafana/grafana:7.3.10
If the running image is older than 8.1.0, patch the ServiceTelemetry object to update the image. Service Telemetry Operator updates the Grafana manifest, which restarts the Grafana deployment:
$ oc patch stf/default --type merge -p '{"spec":{"graphing":{"grafana":{"baseImage":"docker.io/grafana/grafana:8.1.5"}}}}'
Verify that a new Grafana pod exists and has a
STATUS
value ofRunning
:$ oc get pod -l "app=grafana" NAME READY STATUS RESTARTS AGE grafana-deployment-fb9799b58-j2hj2 1/1 Running 0 10s
Verify that the new instance is running the updated image:
$ oc get pod -l "app=grafana" -ojsonpath='{.items[0].spec.containers[0].image}' docker.io/grafana/grafana:8.1.0
5.1.3. Importing dashboards
The Grafana Operator can import and manage dashboards by creating GrafanaDashboard
objects. You can view example dashboards at https://github.com/infrawatch/dashboards.
Procedure
Import the infrastructure dashboard:
$ oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-dashboard.yaml grafanadashboard.integreatly.org/rhos-dashboard-1 created
Import the cloud dashboard:
WarningIn the
stf-connectors.yaml
file, ensure you set the value of the collectdvirt
plugin parameterhostname_format
toname uuid hostname
, otherwise some of the panels on the cloud dashboard display no information. For more information about thevirt
plugin, see collectd plugins.$ oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-cloud-dashboard.yaml grafanadashboard.integreatly.org/rhos-cloud-dashboard-1 created
Import the cloud events dashboard:
$ oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/rhos-cloudevents-dashboard.yaml grafanadashboard.integreatly.org/rhos-cloudevents-dashboard created
Import the virtual machine dashboard:
$ oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/virtual-machine-view.yaml grafanadashboard.integreatly.org/virtual-machine-view-1 configured
Import the memcached dashboard:
$ oc apply -f https://raw.githubusercontent.com/infrawatch/dashboards/master/deploy/stf-1/memcached-dashboard.yaml grafanadashboard.integreatly.org/memcached-dashboard-1 created
Verify that the dashboards are available:
$ oc get grafanadashboards NAME AGE memcached-dashboard-1 7s rhos-cloud-dashboard-1 23s rhos-cloudevents-dashboard 18s rhos-dashboard-1 29s virtual-machine-view-1 13s
Retrieve the Grafana route address:
$ oc get route grafana-route -ojsonpath='{.spec.host}' grafana-route-service-telemetry.apps.infra.watch
- In a web browser, navigate to https://<grafana_route_address>. Replace <grafana_route_address> with the value that you retrieved in the previous step.
- To view the dashboard, click Dashboards and Manage.
5.1.4. Retrieving and setting Grafana login credentials
When Grafana is enabled, you can login using openshift authentication, or the default username and password set by the Grafana Operator.
You can override the credentials in the ServiceTelemetry
object to have Service Telemetry Framework (STF) set the username and password for Grafana instead.
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the
service-telemetry
namespace:$ oc project service-telemetry
Retrieve the existing username and password from the STF object:
$ oc get stf default -o jsonpath="{.spec.graphing.grafana['adminUser','adminPassword']}"
To modify the default values of the Grafana administrator username and password through the ServiceTelemetry object, use the
graphing.grafana.adminUser
andgraphing.grafana.adminPassword
parameters.$ oc edit stf default
Wait for the grafana pod to restart with the new credentials in place
$ oc get po -l app=grafana -w
5.2. Metrics retention time period in Service Telemetry Framework
The default retention time for metrics stored in Service Telemetry Framework (STF) is 24 hours, which provides enough data for trends to develop for the purposes of alerting.
For long-term storage, use systems designed for long-term data retention, for example, Thanos.
Additional resources
- To adjust STF for additional metrics retention time, see Section 5.2.1, “Editing the metrics retention time period in Service Telemetry Framework”.
- For recommendations about Prometheus data storage and estimating storage space, see https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects
- For more information about Thanos, see https://thanos.io/
5.2.1. Editing the metrics retention time period in Service Telemetry Framework
You can adjust Service Telemetry Framework (STF) for additional metrics retention time.
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the service-telemetry namespace:
$ oc project service-telemetry
Edit the ServiceTelemetry object:
$ oc edit stf default
Add
retention: 7d
to the storage section of backends.metrics.prometheus.storage to increase the retention period to seven days:NoteIf you set a long retention period, retrieving data from heavily populated Prometheus systems can result in queries returning results slowly.
apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: ... backends: metrics: prometheus: enabled: true storage: strategy: persistent retention: 7d ...
- Save your changes and close the object.
Wait for prometheus to restart with the new settings.
$ oc get po -l app.kubernetes.io/name=prometheus -w
Verify the new retention setting by checking the command line arguments used in the pod.
$ oc describe po prometheus-default-0 | grep retention.time --storage.tsdb.retention.time=24h
Additional resources
- For more information about the metrics retention time, see Section 5.2, “Metrics retention time period in Service Telemetry Framework”.
5.3. Alerts in Service Telemetry Framework
You create alert rules in Prometheus and alert routes in Alertmanager. Alert rules in Prometheus servers send alerts to an Alertmanager, which manages the alerts. Alertmanager can silence, inhibit, or aggregate alerts, and send notifications by using email, on-call notification systems, or chat platforms.
To create an alert, complete the following tasks:
- Create an alert rule in Prometheus. For more information, see Section 5.3.1, “Creating an alert rule in Prometheus”.
Create an alert route in Alertmanager. There are two ways in which you can create an alert route:
Additional resources
For more information about alerts or notifications with Prometheus and Alertmanager, see https://prometheus.io/docs/alerting/overview/
To view an example set of alerts that you can use with Service Telemetry Framework (STF), see https://github.com/infrawatch/service-telemetry-operator/tree/master/deploy/alerts
5.3.1. Creating an alert rule in Prometheus
Prometheus evaluates alert rules to trigger notifications. If the rule condition returns an empty result set, the condition is false. Otherwise, the rule is true and it triggers an alert.
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the
service-telemetry
namespace:$ oc project service-telemetry
Create a
PrometheusRule
object that contains the alert rule. The Prometheus Operator loads the rule into Prometheus:$ oc apply -f - <<EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: creationTimestamp: null labels: prometheus: default role: alert-rules name: prometheus-alarm-rules namespace: service-telemetry spec: groups: - name: ./openstack.rules rules: - alert: Collectd metrics receive rate is zero expr: rate(sg_total_collectd_msg_received_count[1m]) == 0 EOF
To change the rule, edit the value of the
expr
parameter.To verify that the Operator loaded the rules into Prometheus, run the
curl
command against the default-prometheus-proxy route with basic authentication:$ curl -k --user "internal:$(oc get secret default-prometheus-htpasswd -ogo-template='{{ .data.password | base64decode }}')" https://$(oc get route default-prometheus-proxy -ogo-template='{{ .spec.host }}')/api/v1/rules {"status":"success","data":{"groups":[{"name":"./openstack.rules","file":"/etc/prometheus/rules/prometheus-default-rulefiles-0/service-telemetry-prometheus-alarm-rules.yaml","rules":[{"state":"inactive","name":"Collectd metrics receive count is zero","query":"rate(sg_total_collectd_msg_received_count[1m]) == 0","duration":0,"labels":{},"annotations":{},"alerts":[],"health":"ok","evaluationTime":0.00034627,"lastEvaluation":"2021-12-07T17:23:22.160448028Z","type":"alerting"}],"interval":30,"evaluationTime":0.000353787,"lastEvaluation":"2021-12-07T17:23:22.160444017Z"}]}}
Additional resources
- For more information on alerting, see https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/alerting.md
5.3.2. Configuring custom alerts
You can add custom alerts to the PrometheusRule
object that you created in Section 5.3.1, “Creating an alert rule in Prometheus”.
Procedure
Use the
oc edit
command:$ oc edit prometheusrules prometheus-alarm-rules
-
Edit the
PrometheusRules
manifest. - Save and close the manifest.
Additional resources
- For more information about how to configure alerting rules, see https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/.
- For more information about PrometheusRules objects, see https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/alerting.md
5.3.3. Creating a standard alert route in Alertmanager
Use Alertmanager to deliver alerts to an external system, such as email, IRC, or other notification channel. The Prometheus Operator manages the Alertmanager configuration as a Red Hat OpenShift Container Platform secret. By default, Service Telemetry Framework (STF) deploys a basic configuration that results in no receivers:
alertmanager.yaml: |- global: resolve_timeout: 5m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null'
To deploy a custom Alertmanager route with STF, you must add a alertmanagerConfigManifest
parameter to the Service Telemetry Operator that results in an updated secret, managed by the Prometheus Operator.
If your alertmanagerConfigManifest
contains a custom template, for example, to construct the title and text of the sent alert, you must deploy the contents of the alertmanagerConfigManifest
using a base64-encoded configuration. For more information, see Section 5.3.4, “Creating an alert route with templating in Alertmanager”.
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the
service-telemetry
namespace:$ oc project service-telemetry
Edit the
ServiceTelemetry
object for your STF deployment:$ oc edit stf default
Add the new parameter
alertmanagerConfigManifest
and theSecret
object contents to define thealertmanager.yaml
configuration for Alertmanager:NoteThis step loads the default template that the Service Telemetry Operator manages. To verify that the changes are populating correctly, change a value, return the
alertmanager-default
secret, and verify that the new value is loaded into memory. For example, change the value of the parameterglobal.resolve_timeout
from5m
to10m
.apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: backends: metrics: prometheus: enabled: true alertmanagerConfigManifest: | apiVersion: v1 kind: Secret metadata: name: 'alertmanager-default' namespace: 'service-telemetry' type: Opaque stringData: alertmanager.yaml: |- global: resolve_timeout: 10m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null'
Verify that the configuration has been applied to the secret:
$ oc get secret alertmanager-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}' global: resolve_timeout: 10m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null'
Run the
wget
command from the prometheus pod against thealertmanager-proxy
service to retrieve the status andconfigYAML
contents, and verify that the supplied configuration matches the configuration in Alertmanager:$ oc exec -it prometheus-default-0 -c prometheus -- sh -c "wget --header \"Authorization: Bearer \$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://default-alertmanager-proxy:9095/api/v1/status -q -O -" {"status":"success","data":{"configYAML":"...",...}}
-
Verify that the
configYAML
field contains the changes you expect. To clean up the environment, delete the
curl
pod:$ oc delete pod curl pod "curl" deleted
Additional resources
- For more information about the Red Hat OpenShift Container Platform secret and the Prometheus operator, see Prometheus user guide on alerting.
5.3.4. Creating an alert route with templating in Alertmanager
Use Alertmanager to deliver alerts to an external system, such as email, IRC, or other notification channel. The Prometheus Operator manages the Alertmanager configuration as a Red Hat OpenShift Container Platform secret. By default, Service Telemetry Framework (STF) deploys a basic configuration that results in no receivers:
alertmanager.yaml: |- global: resolve_timeout: 5m route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'null' receivers: - name: 'null'
If the alertmanagerConfigManifest
parameter contains a custom template, for example, to construct the title and text of the sent alert, you must deploy the contents of the alertmanagerConfigManifest
by using a base64-encoded configuration.
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the
service-telemetry
namespace:$ oc project service-telemetry
Create the necessary alertmanager config in a file called alertmanager.yaml, for example:
$ cat > alertmanager.yaml <<EOF global: resolve_timeout: 10m slack_api_url: <slack_api_url> receivers: - name: slack slack_configs: - channel: #stf-alerts title: |- ... text: >- ... route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'slack' EOF
Generate the config manifest and add it to the
ServiceTelemetry
object for your STF deployment:$ CONFIG_MANIFEST=$(oc create secret --dry-run=client generic alertmanager-default --from-file=alertmanager.yaml -o json) $ oc patch stf default --type=merge -p '{"spec":{"alertmanagerConfigManifest":'"$CONFIG_MANIFEST"'}}'
Verify that the configuration has been applied to the secret:
NoteThere will be a short delay as the operators update each object
$ oc get secret alertmanager-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}' global: resolve_timeout: 10m slack_api_url: <slack_api_url> receivers: - name: slack slack_configs: - channel: #stf-alerts title: |- ... text: >- ... route: group_by: ['job'] group_wait: 30s group_interval: 5m repeat_interval: 12h receiver: 'slack'
Run the
wget
command from the prometheus pod against thealertmanager-proxy
service to retrieve the status andconfigYAML
contents, and verify that the supplied configuration matches the configuration in Alertmanager:$ oc exec -it prometheus-default-0 -c prometheus -- /bin/sh -c "wget --header \"Authorization: Bearer \$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://default-alertmanager-proxy:9095/api/v1/status -q -O -" {"status":"success","data":{"configYAML":"...",...}}
-
Verify that the
configYAML
field contains the changes you expect.
Additional resources
- For more information about the Red Hat OpenShift Container Platform secret and the Prometheus operator, see Prometheus user guide on alerting.
5.4. Sending alerts as SNMP traps
To enable SNMP traps, modify the ServiceTelemetry
object and configure the snmpTraps
parameters. SNMP traps are sent using version 2c.
5.4.1. Configuration parameters for snmpTraps
The snmpTraps
parameter contains the following sub-parameters for configuring the alert receiver:
- enabled
- Set the value of this sub-parameter to true to enable the SNMP trap alert receiver. The default value is false.
- target
-
Target address to send SNMP traps. Value is a string. Default is
192.168.24.254
. - port
-
Target port to send SNMP traps. Value is an integer. Default is
162
. - community
-
Target community to send SNMP traps to. Value is a string. Default is
public
. - retries
-
SNMP trap retry delivery limit. Value is an integer. Default is
5
. - timeout
-
SNMP trap delivery timeout defined in seconds. Value is an integer. Default is
1
. - alertOidLabel
-
Label name in the alert that defines the OID value to send the SNMP trap as. Value is a string. Default is
oid
. - trapOidPrefix
-
SNMP trap OID prefix for variable bindings. Value is a string. Default is
1.3.6.1.4.1.50495.15
. - trapDefaultOid
-
SNMP trap OID when no alert OID label has been specified with the alert. Value is a string. Default is
1.3.6.1.4.1.50495.15.1.2.1
. - trapDefaultSeverity
- SNMP trap severity when no alert severity has been set. Value is a string. Defaults to an empty string.
Configure the snmpTraps
parameter as part of the alerting.alertmanager.receivers
definition in the ServiceTelemetry
object:
apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: alerting: alertmanager: receivers: snmpTraps: alertOidLabel: oid community: public enabled: true port: 162 retries: 5 target: 192.168.25.254 timeout: 1 trapDefaultOid: 1.3.6.1.4.1.50495.15.1.2.1 trapDefaultSeverity: "" trapOidPrefix: 1.3.6.1.4.1.50495.15 ...
5.4.2. Overview of the MIB definition
Delivery of SNMP traps uses object identifier (OID) value 1.3.6.1.4.1.50495.15.1.2.1
by default. The management information base (MIB) schema is available at https://github.com/infrawatch/prometheus-webhook-snmp/blob/master/PROMETHEUS-ALERT-CEPH-MIB.txt.
The OID number is comprised of the following component values: * The value 1.3.6.1.4.1
is a global OID defined for private enterprises. * The next identifier 50495
is a private enterprise number assigned by IANA for the Ceph organization. * The other values are child OIDs of the parent.
- 15
- prometheus objects
- 15.1
- prometheus alerts
- 15.1.2
- prometheus alert traps
- 15.1.2.1
- prometheus alert trap default
The prometheus alert trap default is an object comprised of several other sub-objects to OID 1.3.6.1.4.1.50495.15
which is defined by the alerting.alertmanager.receivers.snmpTraps.trapOidPrefix
parameter:
- <trapOidPrefix>.1.1.1
- alert name
- <trapOidPrefix>.1.1.2
- status
- <trapOidPrefix>.1.1.3
- severity
- <trapOidPrefix>.1.1.4
- instance
- <trapOidPrefix>.1.1.5
- job
- <trapOidPrefix>.1.1.6
- description
- <trapOidPrefix>.1.1.7
- labels
- <trapOidPrefix>.1.1.8
- timestamp
- <trapOidPrefix>.1.1.9
- rawdata
The following is example output from a simple SNMP trap receiver that outputs the received trap to the console:
SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.50495.15.1.2.1 SNMPv2-SMI::enterprises.50495.15.1.1.1 = STRING: "TEST ALERT FROM PROMETHEUS PLEASE ACKNOWLEDGE" SNMPv2-SMI::enterprises.50495.15.1.1.2 = STRING: "firing" SNMPv2-SMI::enterprises.50495.15.1.1.3 = STRING: "warning" SNMPv2-SMI::enterprises.50495.15.1.1.4 = "" SNMPv2-SMI::enterprises.50495.15.1.1.5 = "" SNMPv2-SMI::enterprises.50495.15.1.1.6 = STRING: "TEST ALERT FROM " SNMPv2-SMI::enterprises.50495.15.1.1.7 = STRING: "{\"cluster\": \"TEST\", \"container\": \"sg-core\", \"endpoint\": \"prom-https\", \"prometheus\": \"service-telemetry/default\", \"service\": \"default-cloud1-coll-meter\", \"source\": \"SG\"}" SNMPv2-SMI::enterprises.50495.15.1.1.8 = Timeticks: (1676476389) 194 days, 0:52:43.89 SNMPv2-SMI::enterprises.50495.15.1.1.9 = STRING: "{\"status\": \"firing\", \"labels\": {\"cluster\": \"TEST\", \"container\": \"sg-core\", \"endpoint\": \"prom-https\", \"prometheus\": \"service-telemetry/default\", \"service\": \"default-cloud1-coll-meter\", \"source\": \"SG\"}, \"annotations\": {\"action\": \"TESTING PLEASE ACKNOWLEDGE, NO FURTHER ACTION REQUIRED ONLY A TEST\"}, \"startsAt\": \"2023-02-15T15:53:09.109Z\", \"endsAt\": \"0001-01-01T00:00:00Z\", \"generatorURL\": \"http://prometheus-default-0:9090/graph?g0.expr=sg_total_collectd_msg_received_count+%3E+1&g0.tab=1\", \"fingerprint\": \"feefeb77c577a02f\"}"
5.4.3. Configuring SNMP traps
Prerequisites
- Ensure that you know the IP address or hostname of the SNMP trap receiver where you want to send the alerts to.
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the
service-telemetry
namespace:$ oc project service-telemetry
To enable SNMP traps, modify the
ServiceTelemetry
object:$ oc edit stf default
Set the
alerting.alertmanager.receivers.snmpTraps
parameters:apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry ... spec: ... alerting: alertmanager: receivers: snmpTraps: enabled: true target: 10.10.10.10
-
Ensure that you set the value of
target
to the IP address or hostname of the SNMP trap receiver.
Additional Information
For more information about available parameters for snmpTraps
, see Section 5.4.1, “Configuration parameters for snmpTraps”.
5.4.4. Creating alerts for SNMP traps
You can create alerts that are configured for delivery by SNMP traps by adding labels that are parsed by the prometheus-webhook-snmp middleware to define the trap information and delivered object identifiers (OID). Adding the oid
or severity
labels is only required if you need to change the default values for a particular alert definition.
- NOTE
-
When you set the oid label, the top-level SNMP trap OID changes, but the sub-OIDs remain defined by the global
trapOidPrefix
value plus the child OID values.1.1.1
through.1.1.9
. For more information about the MIB definition, see Section 5.4.2, “Overview of the MIB definition”.
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the
service-telemetry
namespace:$ oc project service-telemetry
Create a
PrometheusRule
object that contains the alert rule and anoid
label that contains the SNMP trap OID override value:$ oc apply -f - <<EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: creationTimestamp: null labels: prometheus: default role: alert-rules name: prometheus-alarm-rules-snmp namespace: service-telemetry spec: groups: - name: ./openstack.rules rules: - alert: Collectd metrics receive rate is zero expr: rate(sg_total_collectd_msg_received_count[1m]) == 0 labels: oid: 1.3.6.1.4.1.50495.15.1.2.1 severity: critical EOF
Additional information
For more information about configuring alerts, see Section 5.3, “Alerts in Service Telemetry Framework”.
5.5. High availability
With high availability, Service Telemetry Framework (STF) can rapidly recover from failures in its component services. Although Red Hat OpenShift Container Platform restarts a failed pod if nodes are available to schedule the workload, this recovery process might take more than one minute, during which time events and metrics are lost. A high availability configuration includes multiple copies of STF components, which reduces recovery time to approximately 2 seconds. To protect against failure of an Red Hat OpenShift Container Platform node, deploy STF to an Red Hat OpenShift Container Platform cluster with three or more nodes.
STF is not yet a fully fault tolerant system. Delivery of metrics and events during the recovery period is not guaranteed.
Enabling high availability has the following effects:
- Three Elasticsearch pods run instead of the default one.
The following components run two pods instead of the default one:
- AMQ Interconnect
- Alertmanager
- Prometheus
- Events Smart Gateway
- Metrics Smart Gateway
- Recovery time from a lost pod in any of these services reduces to approximately 2 seconds.
5.5.1. Configuring high availability
To configure Service Telemetry Framework (STF) for high availability, add highAvailability.enabled: true
to the ServiceTelemetry object in Red Hat OpenShift Container Platform. You can set this parameter at installation time or, if you already deployed STF, complete the following steps:
Procedure
- Log in to Red Hat OpenShift Container Platform.
Change to the
service-telemetry
namespace:$ oc project service-telemetry
Use the oc command to edit the ServiceTelemetry object:
$ oc edit stf default
Add
highAvailability.enabled: true
to thespec
section:apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry ... spec: ... highAvailability: enabled: true
- Save your changes and close the object.
5.6. Observability Strategy in Service Telemetry Framework
Service Telemetry Framework (STF) does not include storage backends and alerting tools. STF uses community operators to deploy Prometheus, Alertmanager, Grafana, and Elasticsearch. STF makes requests to these community operators to create instances of each application configured to work with STF.
Instead of having Service Telemetry Operator create custom resource requests, you can use your own deployments of these applications or other compatible applications, and scrape the metrics Smart Gateways for delivery to your own Prometheus-compatible system for telemetry storage. If you set the observabilityStrategy
to none
, then storage backends will not be deployed so persistent storage will not be required by STF.
5.6.1. Configuring an alternate observability strategy
To configure STF to skip the deployment of storage, visualization, and alerting backends, add observabilityStrategy: none
to the ServiceTelemetry spec. In this mode, only AMQ Interconnect routers and metrics Smart Gateways are deployed, and you must configure an external Prometheus-compatible system to collect metrics from the STF Smart Gateways.
Currently, only metrics are supported when you set observabilityStrategy
to none
. Events Smart Gateways are not deployed.
Procedure
Create a
ServiceTelemetry
object with the propertyobservabilityStrategy: none
in thespec
parameter. The manifest shows results in a default deployment of STF that is suitable for receiving telemetry from a single cloud with all metrics collector types.$ oc apply -f - <<EOF apiVersion: infra.watch/v1beta1 kind: ServiceTelemetry metadata: name: default namespace: service-telemetry spec: observabilityStrategy: none EOF
Delete the left over objects that are managed by community operators
$ for o in alertmanager/default prometheus/default elasticsearch/elasticsearch grafana/default; do oc delete $o; done
To verify that all workloads are operating correctly, view the pods and the status of each pod:
$ oc get pods NAME READY STATUS RESTARTS AGE default-cloud1-ceil-meter-smartgateway-59c845d65b-gzhcs 3/3 Running 0 132m default-cloud1-coll-meter-smartgateway-75bbd948b9-d5phm 3/3 Running 0 132m default-cloud1-sens-meter-smartgateway-7fdbb57b6d-dh2g9 3/3 Running 0 132m default-interconnect-668d5bbcd6-57b2l 1/1 Running 0 132m interconnect-operator-b8f5bb647-tlp5t 1/1 Running 0 47h service-telemetry-operator-566b9dd695-wkvjq 1/1 Running 0 156m smart-gateway-operator-58d77dcf7-6xsq7 1/1 Running 0 47h
Additional resources
For more information about configuring additional clouds or to change the set of supported collectors, see Section 4.3.2, “Deploying Smart Gateways”
5.7. Resource usage of Red Hat OpenStack Platform services
You can monitor the resource usage of the Red Hat OpenStack Platform (RHOSP) services, such as the APIs and other infrastructure processes, to identify bottlenecks in the overcloud by showing services that run out of compute power. Resource usage monitoring is enabled by default.
Additional resources
- To disable resource usage monitoring, see Section 5.7.1, “Disabling resource usage monitoring of Red Hat OpenStack Platform services”.
5.7.1. Disabling resource usage monitoring of Red Hat OpenStack Platform services
To disable the monitoring of RHOSP containerized service resource usage, you must set the CollectdEnableLibpodstats
parameter to false
.
Prerequisites
-
You have created the
stf-connectors.yaml
file. For more information, see Section 4.1, “Deploying Red Hat OpenStack Platform overcloud for Service Telemetry Framework using director”. - You are using the most current version of Red Hat OpenStack Platform (RHOSP) 16.1.
Procedure
Open the
stf-connectors.yaml
file and add theCollectdEnableLibpodstats
parameter to override the setting inenable-stf.yaml
. Ensure thatstf-connectors.yaml
is called from theopenstack overcloud deploy
command afterenable-stf.yaml
:CollectdEnableLibpodstats: false
- Continue with the overcloud deployment procedure. For more information, see Section 4.1.4, “Deploying the overcloud”.
5.8. Red Hat OpenStack Platform API status and containerized services health
You can use the OCI (Open Container Initiative) standard to assess the container health status of each Red Hat OpenStack Platform (RHOSP) service by periodically running a health check script. Most RHOSP services implement a health check that logs issues and returns a binary status. For the RHOSP APIs, the health checks query the root endpoint and determine the health based on the response time.
Monitoring of RHOSP container health and API status is enabled by default.
Additional resources
- To disable RHOSP container health and API status monitoring, see Section 5.8.1, “Disabling container health and API status monitoring”.
5.8.1. Disabling container health and API status monitoring
To disable RHOSP containerized service health and API status monitoring, you must set the CollectdEnableSensubility
parameter to false
.
Prerequisites
-
You have created the
stf-connectors.yaml
file in your templates directory. For more information, see Section 4.1, “Deploying Red Hat OpenStack Platform overcloud for Service Telemetry Framework using director”. - You are using the most current version of Red Hat OpenStack Platform (RHOSP) 16.1.
Procedure
Open the
stf-connectors.yaml
and add theCollectdEnableSensubility
parameter to override the setting inenable-stf.yaml
. Ensure thatstf-connectors.yaml
is called from theopenstack overcloud deploy
command afterenable-stf.yaml
:CollectdEnableSensubility: false
- Continue with the overcloud deployment procedure. For more information, see Section 4.1.4, “Deploying the overcloud”.
Additional resources
- For more information about multiple cloud addresses, see Section 4.3, “Configuring multiple clouds”.