Chapter 6. Using operational features of Service Telemetry Framework

Procedure

1. Log in to your Red Hat OpenShift Container Platform environment where STF is hosted.

2. Subscribe to the Grafana Operator by using the community-operators CatalogSource:

   Warning

   Community Operators are Operators which have not been vetted or verified by Red Hat. Community Operators should be used with caution because their stability is unknown. Red Hat provides no support for community Operators.

   Learn more about Red Hat’s third party software support policy

   $ oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     labels:
       operators.coreos.com/grafana-operator.openshift-operators: ""
     name: grafana-operator
     namespace: openshift-operators
   spec:
     channel: v5
     installPlanApproval: Automatic
     name: grafana-operator
     source: community-operators
     sourceNamespace: openshift-marketplace
   EOF

   Copy to Clipboard Toggle word wrap

3. Verify that the Operator launched successfully. In the command output, if the value of the PHASE column is Succeeded, the Operator launched successfully:

   $ oc wait --for jsonpath="{.status.phase}"=Succeeded csv --namespace openshift-operators -l operators.coreos.com/grafana-operator.openshift-operators
   
   clusterserviceversion.operators.coreos.com/grafana-operator.v5.6.0 condition met

   Copy to Clipboard Toggle word wrap

4. To launch a Grafana instance, create or modify the ServiceTelemetry object. Set graphing.enabled and graphing.grafana.ingressEnabled to true. Optionally, set the value of graphing.grafana.baseImage to the Grafana workload container image that will be deployed:

   $ oc edit stf default
   
   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   ...
   spec:
     ...
     graphing:
       enabled: true
       grafana:
         ingressEnabled: true
         baseImage: 'registry.redhat.io/rhel8/grafana:9'

   Copy to Clipboard Toggle word wrap

5. Verify that the Grafana instance deployed:

   $ oc wait --for jsonpath="{.status.phase}"=Running pod -l app=default-grafana --timeout=600s
   
   pod/default-grafana-deployment-669968df64-wz5s2 condition met

   Copy to Clipboard Toggle word wrap

6. Verify that the Grafana data sources installed correctly:

   $ oc get grafanadatasources.grafana.integreatly.org
   
   NAME                        NO MATCHING INSTANCES   LAST RESYNC   AGE
   default-ds-stf-prometheus                           2m35s         2m56s

   Copy to Clipboard Toggle word wrap

7. Verify that the Grafana route exists:

   $ oc get route default-grafana-route
   
   NAME                    HOST/PORT                                                   PATH   SERVICES                  PORT   TERMINATION   WILDCARD
   default-grafana-route   default-grafana-route-service-telemetry.apps.infra.watch           default-grafana-service   web    reencrypt     None

   Copy to Clipboard Toggle word wrap

Procedure

1. To enable the managed dashboards, create or modify the ServiceTelemetry object. Set graphing.grafana.dashboards.enabled to true:

   $ oc edit stf default
   
   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   ...
   spec:
     ...
     graphing:
       enabled: true
       grafana:
         dashboards:
           enabled: true

   Copy to Clipboard Toggle word wrap

2. Verify that the Grafana dashboards are created. The process of Service Telemetry Operator creating the dashboards might take some time.

   $ oc get grafanadashboards.grafana.integreatly.org
   
   NAME                          NO MATCHING INSTANCES   LAST RESYNC   AGE
   memcached-dashboard-1                                 38s           38s
   rhos-cloud-dashboard-1                                39s           39s
   rhos-dashboard-1                                      39s           39s
   virtual-machine-dashboard-1                           37s           37s

   Copy to Clipboard Toggle word wrap

3. Retrieve the Grafana route address:

   $ oc get route default-grafana-route -ojsonpath='{.spec.host}'
   
   default-grafana-route-service-telemetry.apps.infra.watch

   Copy to Clipboard Toggle word wrap
4. In a web browser, navigate to https://<grafana_route_address>. Replace <grafana_route_address> with the value that you retrieved in the previous step.
5. Log in with OpenShift credentials. For more information about logging in, see Section 3.3, “Accessing user interfaces for STF components”.
6. To view the dashboard, click Dashboards and Browse. The managed dashboards are available in the service-telemetry folder.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Create a new token secret for the stf-prometheus-reader service account

   $ oc create -f - <<EOF
   apiVersion: v1
   kind: Secret
   metadata:
     name: my-prometheus-reader-token
     namespace: service-telemetry
     annotations:
       kubernetes.io/service-account.name: stf-prometheus-reader
   type: kubernetes.io/service-account-token
   EOF

   Copy to Clipboard Toggle word wrap

4. Retrieve the token from the secret

   $ TOKEN=$(oc get secret my-prometheus-reader-token -o template='{{.data.token}}' | base64 -d)

   Copy to Clipboard Toggle word wrap

5. Retrieve the Prometheus host name

   $ PROM_HOST=$(oc get route default-prometheus-proxy -ogo-template='{{ .spec.host }}')

   Copy to Clipboard Toggle word wrap

6. Test the access token

   $ curl -k -H "Authorization: Bearer ${TOKEN}" https://${PROM_HOST}/api/v1/query?query=up
   
   {"status":"success",[...]

   Copy to Clipboard Toggle word wrap

7. Configure your third-party tool with the PROM_HOST and TOKEN values from above

   $ echo $PROM_HOST
   $ echo $TOKEN

   Copy to Clipboard Toggle word wrap

8. The token remains valid as long as the secret exists. You can revoke the token by deleting the secret.

   $ oc delete secret my-prometheus-reader-token
   secret "my-prometheus-reader-token" deleted

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Edit the ServiceTelemetry object:

   $ oc edit stf default

   Copy to Clipboard Toggle word wrap

4. Add retention: 7d to the storage section of backends.metrics.prometheus.storage to increase the retention period to seven days:

   Note

   If you set a long retention period, retrieving data from heavily populated Prometheus systems can result in queries returning results slowly.

   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   metadata:
     name: default
     namespace: service-telemetry
   spec:
     ...
     backends:
       metrics:
         prometheus:
           enabled: true
           storage:
             strategy: persistent
             retention: 7d
       ...

   Copy to Clipboard Toggle word wrap
5. Save your changes and close the object.

6. Wait for prometheus to restart with the new settings.

   $ oc get po -l app.kubernetes.io/name=prometheus -w

   Copy to Clipboard Toggle word wrap

7. Verify the new retention setting by checking the command line arguments used in the pod.

   $ oc describe po prometheus-default-0 | grep retention.time
         --storage.tsdb.retention.time=24h

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Create a PrometheusRule object that contains the alert rule. The Prometheus Operator loads the rule into Prometheus:

   $ oc apply -f - <<EOF
   apiVersion: monitoring.rhobs/v1
   kind: PrometheusRule
   metadata:
     creationTimestamp: null
     labels:
       prometheus: default
       role: alert-rules
     name: prometheus-alarm-rules
     namespace: service-telemetry
   spec:
     groups:
       - name: ./openstack.rules
         rules:
           - alert: Collectd metrics receive rate is zero
             expr: rate(sg_total_collectd_msg_received_count[1m]) == 0
   EOF

   Copy to Clipboard Toggle word wrap

   To change the rule, edit the value of the expr parameter.

4. To verify that the Operator loaded the rules into Prometheus, run the curl command against the default-prometheus-proxy route with basic authentication:

   $ curl -k -H "Authorization: Bearer $(oc create token stf-prometheus-reader)" https://$(oc get route default-prometheus-proxy -ogo-template='{{ .spec.host }}')/api/v1/rules
   
   {"status":"success","data":{"groups":[{"name":"./openstack.rules","file":"/etc/prometheus/rules/prometheus-default-rulefiles-0/service-telemetry-prometheus-alarm-rules.yaml","rules":[{"state":"inactive","name":"Collectd metrics receive count is zero","query":"rate(sg_total_collectd_msg_received_count[1m]) == 0","duration":0,"labels":{},"annotations":{},"alerts":[],"health":"ok","evaluationTime":0.00034627,"lastEvaluation":"2021-12-07T17:23:22.160448028Z","type":"alerting"}],"interval":30,"evaluationTime":0.000353787,"lastEvaluation":"2021-12-07T17:23:22.160444017Z"}]}}

   Copy to Clipboard Toggle word wrap

Procedure

1. Use the oc edit command:

   $ oc edit prometheusrules.monitoring.rhobs prometheus-alarm-rules

   Copy to Clipboard Toggle word wrap
2. Edit the PrometheusRules manifest.
3. Save and close the manifest.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Edit the ServiceTelemetry object for your STF deployment:

   $ oc edit stf default

   Copy to Clipboard Toggle word wrap

4. Add the new parameter alertmanagerConfigManifest and the Secret object contents to define the alertmanager.yaml configuration for Alertmanager:

   Note

   This step loads the default template that the Service Telemetry Operator manages. To verify that the changes are populating correctly, change a value, return the alertmanager-default secret, and verify that the new value is loaded into memory. For example, change the value of the parameter global.resolve_timeout from 5m to 10m.

   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   metadata:
     name: default
     namespace: service-telemetry
   spec:
     backends:
       metrics:
         prometheus:
           enabled: true
     alertmanagerConfigManifest: |
       apiVersion: v1
       kind: Secret
       metadata:
         name: 'alertmanager-default'
         namespace: 'service-telemetry'
       type: Opaque
       stringData:
         alertmanager.yaml: |-
           global:
             resolve_timeout: 10m
           route:
             group_by: ['job']
             group_wait: 30s
             group_interval: 5m
             repeat_interval: 12h
             receiver: 'null'
           receivers:
           - name: 'null'

   Copy to Clipboard Toggle word wrap

5. Verify that the configuration has been applied to the secret:

   $ oc get secret alertmanager-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}'
   
   global:
     resolve_timeout: 10m
   route:
     group_by: ['job']
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: 'null'
   receivers:
   - name: 'null'

   Copy to Clipboard Toggle word wrap

6. Run the wget command from the prometheus pod against the alertmanager-proxy service to retrieve the status and configYAML contents, and verify that the supplied configuration matches the configuration in Alertmanager:

   $ oc exec -it prometheus-default-0 -c prometheus -- sh -c "wget --header \"Authorization: Bearer \$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://default-alertmanager-proxy:9095/api/v1/status -q -O -"
   
   {"status":"success","data":{"configYAML":"...",...}}

   Copy to Clipboard Toggle word wrap
7. Verify that the configYAML field contains the changes you expect.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Create the necessary alertmanager config in a file called alertmanager.yaml, for example:

   $ cat > alertmanager.yaml <<EOF
   global:
     resolve_timeout: 10m
     slack_api_url: <slack_api_url>
   receivers:
     - name: slack
       slack_configs:
       - channel: #stf-alerts
         title: |-
           ...
         text: >-
           ...
   route:
     group_by: ['job']
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: 'slack'
   EOF

   Copy to Clipboard Toggle word wrap

4. Generate the config manifest and add it to the ServiceTelemetry object for your STF deployment:

   $ CONFIG_MANIFEST=$(oc create secret --dry-run=client generic alertmanager-default --from-file=alertmanager.yaml -o json)
   $ oc patch stf default --type=merge -p '{"spec":{"alertmanagerConfigManifest":'"$CONFIG_MANIFEST"'}}'

   Copy to Clipboard Toggle word wrap

5. Verify that the configuration has been applied to the secret:

   Note

   There will be a short delay as the operators update each object

   $ oc get secret alertmanager-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}'
   
   global:
     resolve_timeout: 10m
     slack_api_url: <slack_api_url>
   receivers:
     - name: slack
       slack_configs:
       - channel: #stf-alerts
         title: |-
           ...
         text: >-
           ...
   route:
     group_by: ['job']
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: 'slack'

   Copy to Clipboard Toggle word wrap

6. Run the wget command from the prometheus pod against the alertmanager-proxy service to retrieve the status and configYAML contents, and verify that the supplied configuration matches the configuration in Alertmanager:

   $ oc exec -it prometheus-default-0 -c prometheus -- /bin/sh -c "wget --header \"Authorization: Bearer \$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://default-alertmanager-proxy:9095/api/v1/status -q -O -"
   
   {"status":"success","data":{"configYAML":"...",...}}

   Copy to Clipboard Toggle word wrap
7. Verify that the configYAML field contains the changes you expect.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. To enable SNMP traps, modify the ServiceTelemetry object:

   $ oc edit stf default

   Copy to Clipboard Toggle word wrap

4. Set the alerting.alertmanager.receivers.snmpTraps parameters:

   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   ...
   spec:
     ...
     alerting:
       alertmanager:
         receivers:
           snmpTraps:
             enabled: true
             target: 10.10.10.10

   Copy to Clipboard Toggle word wrap
5. Ensure that you set the value of target to the IP address or hostname of the SNMP trap receiver.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Create a PrometheusRule object that contains the alert rule and an oid label that contains the SNMP trap OID override value:

   $ oc apply -f - <<EOF
   apiVersion: monitoring.rhobs/v1
   kind: PrometheusRule
   metadata:
     creationTimestamp: null
     labels:
       prometheus: default
       role: alert-rules
     name: prometheus-alarm-rules-snmp
     namespace: service-telemetry
   spec:
     groups:
       - name: ./openstack.rules
         rules:
           - alert: Collectd metrics receive rate is zero
             expr: rate(sg_total_collectd_msg_received_count[1m]) == 0
             labels:
               oid: 1.3.6.1.4.1.50495.15.1.2.1
               severity: critical
   EOF

   Copy to Clipboard Toggle word wrap

Procedure

1. To edit the duration of the TLS certificate, you can set the QDR caCertDuration, for example 87600h for 10 years:

   $ oc apply -f - <<EOF
   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   metadata:
     name: default
     namespace: service-telemetry
   spec:
     transport:
       qdr:
         enabled: true
         certificates:
           caCertDuration: 87600h
   EOF

   Copy to Clipboard Toggle word wrap

Verification

1. Verify that the expiry date for the certificate is correct:

   $ oc get secret  default-interconnect-selfsigned -o jsonpath='{.data.tls\.crt}' | base64 -d  | openssl x509 -in - -text | grep "Not After"
               Not After : Mar  9 21:00:16 2033 GMT

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Use the oc command to edit the ServiceTelemetry object:

   $ oc edit stf default

   Copy to Clipboard Toggle word wrap

4. Add highAvailability.enabled: true to the spec section:

   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   ...
   spec:
     ...
     highAvailability:
       enabled: true

   Copy to Clipboard Toggle word wrap
5. Save your changes and close the object.

Procedure

1. Create a ServiceTelemetry object with the property observabilityStrategy: none in the spec parameter. The manifest shows results in a default deployment of STF that is suitable for receiving telemetry from a single cloud with all metrics collector types.

   $ oc apply -f - <<EOF
   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   metadata:
     name: default
     namespace: service-telemetry
   spec:
     observabilityStrategy: none
   EOF

   Copy to Clipboard Toggle word wrap

2. Delete the remaining objects that are managed by community operators

   $ for o in alertmanagers.monitoring.rhobs/default prometheuses.monitoring.rhobs/default elasticsearch/elasticsearch grafana/default-grafana; do oc delete $o; done

   Copy to Clipboard Toggle word wrap

3. To verify that all workloads are operating correctly, view the pods and the status of each pod:

   $ oc get pods
   NAME                                                      READY   STATUS    RESTARTS   AGE
   default-cloud1-ceil-event-smartgateway-6f8547df6c-p2db5   3/3     Running   0          132m
   default-cloud1-ceil-meter-smartgateway-59c845d65b-gzhcs   3/3     Running   0          132m
   default-cloud1-coll-event-smartgateway-bf859f8d77-tzb66   3/3     Running   0          132m
   default-cloud1-coll-meter-smartgateway-75bbd948b9-d5phm   3/3     Running   0          132m
   default-cloud1-sens-meter-smartgateway-7fdbb57b6d-dh2g9   3/3     Running   0          132m
   default-interconnect-668d5bbcd6-57b2l                     1/1     Running   0          132m
   interconnect-operator-b8f5bb647-tlp5t                     1/1     Running   0          47h
   service-telemetry-operator-566b9dd695-wkvjq               1/1     Running   0          156m
   smart-gateway-operator-58d77dcf7-6xsq7                    1/1     Running   0          47h

   Copy to Clipboard Toggle word wrap

Procedure

1. Open the stf-connectors.yaml file and add the CollectdEnableLibpodstats parameter to override the setting in enable-stf.yaml. Ensure that stf-connectors.yaml is called from the openstack overcloud deploy command after enable-stf.yaml:

     CollectdEnableLibpodstats: false

   Copy to Clipboard Toggle word wrap
2. Continue with the overcloud deployment procedure. For more information, see Section 4.1.6, “Deploying the overcloud”.

Procedure

1. Open the stf-connectors.yaml and add the CollectdEnableSensubility parameter to override the setting in enable-stf.yaml. Ensure that stf-connectors.yaml is called from the openstack overcloud deploy command after enable-stf.yaml:

   CollectdEnableSensubility: false

   Copy to Clipboard Toggle word wrap
2. Continue with the overcloud deployment procedure. For more information, see Section 4.1.6, “Deploying the overcloud”.