Chapter 5. Using operational features of Service Telemetry Framework

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Deploy the Grafana operator:

   $ oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: grafana-operator
     namespace: service-telemetry
   spec:
     channel: alpha
     installPlanApproval: Automatic
     name: grafana-operator
     source: operatorhubio-operators
     sourceNamespace: openshift-marketplace
   EOF

   Copy to Clipboard Toggle word wrap

4. Verify that the Operator launched successfully. In the command output, if the value of the PHASE column is Succeeded, the Operator launched successfully:

   $ oc get csv --selector operators.coreos.com/grafana-operator.service-telemetry
   
   NAME                       DISPLAY            VERSION   REPLACES                   PHASE
   grafana-operator.v3.10.3   Grafana Operator   3.10.3    grafana-operator.v3.10.2   Succeeded

   Copy to Clipboard Toggle word wrap

5. To launch a Grafana instance, create or modify the ServiceTelemetry object. Set graphing.enabled and graphing.grafana.ingressEnabled to true:

   $ oc edit stf default
   
   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   ...
   spec:
     ...
     graphing:
       enabled: true
       grafana:
         ingressEnabled: true

   Copy to Clipboard Toggle word wrap

6. Verify that the Grafana instance deployed:

   $ oc get pod -l app=grafana
   
   NAME                                  READY   STATUS    RESTARTS   AGE
   grafana-deployment-7fc7848b56-sbkhv   1/1     Running   0          1m

   Copy to Clipboard Toggle word wrap

7. Verify that the Grafana data sources installed correctly:

   $ oc get grafanadatasources
   
   NAME                    AGE
   default-datasources     20h

   Copy to Clipboard Toggle word wrap

8. Verify that the Grafana route exists:

   $ oc get route grafana-route
   
   NAME            HOST/PORT                                          PATH   SERVICES          PORT   TERMINATION   WILDCARD
   grafana-route   grafana-route-service-telemetry.apps.infra.watch          grafana-service   3000   edge          None

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. To retrieve the default username and password, describe the Grafana object:

   $ oc describe grafana default

   Copy to Clipboard Toggle word wrap
4. To modify the default values of the Grafana administrator username and password through the ServiceTelemetry object, use the graphing.grafana.adminUser and graphing.grafana.adminPassword parameters.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Edit the ServiceTelemetry object:

   $ oc edit stf default

   Copy to Clipboard Toggle word wrap

4. Add retention: 7d to the storage section of backends.metrics.prometheus.storage to increase the retention period to seven days:

   Note

   If you set a long retention period, retrieving data from heavily populated Prometheus systems can result in queries returning results slowly.

   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   metadata:
     name: stf-default
     namespace: service-telemetry
   spec:
     ...
     backends:
       metrics:
         prometheus:
           enabled: true
           storage:
             strategy: persistent
             retention: 7d
       ...

   Copy to Clipboard Toggle word wrap
5. Save your changes and close the object.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Create a PrometheusRule object that contains the alert rule. The Prometheus Operator loads the rule into Prometheus:

   $ oc apply -f - <<EOF
   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     creationTimestamp: null
     labels:
       prometheus: default
       role: alert-rules
     name: prometheus-alarm-rules
     namespace: service-telemetry
   spec:
     groups:
       - name: ./openstack.rules
         rules:
           - alert: Metric Listener down
             expr: collectd_qpid_router_status < 1 

   1

   
   EOF

   Copy to Clipboard Toggle word wrap

   1

   To change the rule, edit the value of the expr parameter.

4. To verify that the Operator loaded the rules into Prometheus, create a pod with access to curl:

   $ oc run curl --generator=run-pod/v1 --image=radial/busyboxplus:curl -i --tty

   Copy to Clipboard Toggle word wrap

5. Run the curl command to access the prometheus-operated service to return the rules loaded into memory:

   [ root@curl:/ ]$ curl prometheus-operated:9090/api/v1/rules
   {"status":"success","data":{"groups":[{"name":"./openstack.rules","file":"/etc/prometheus/rules/prometheus-default-rulefiles-0/service-telemetry-prometheus-alarm-rules.yaml","rules":[{"name":"Metric Listener down","query":"collectd_qpid_router_status \u003c 1","duration":0,"labels":{},"annotations":{},"alerts":[],"health":"ok","type":"alerting"}],"interval":30}]}}

   Copy to Clipboard Toggle word wrap

6. To verify that the output shows the rules loaded into the PrometheusRule object, for example the output contains the defined ./openstack.rules, exit the pod:

   [ root@curl:/ ]$ exit

   Copy to Clipboard Toggle word wrap

7. Clean up the environment by deleting the curl pod:

   $ oc delete pod curl
   
   pod "curl" deleted

   Copy to Clipboard Toggle word wrap

Procedure

1. Use the oc edit command:

   $ oc edit prometheusrules prometheus-alarm-rules

   Copy to Clipboard Toggle word wrap
2. Edit the PrometheusRules manifest.
3. Save and close the manifest.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Edit the ServiceTelemetry object for your STF deployment:

   $ oc edit stf default

   Copy to Clipboard Toggle word wrap

4. Add the new parameter alertmanagerConfigManifest and the Secret object contents to define the alertmanager.yaml configuration for Alertmanager:

   Note

   This step loads the default template that the Service Telemetry Operator manages. To verify that the changes are populating correctly, change a value, return the alertmanager-default secret, and verify that the new value is loaded into memory. For example, change the value of the parameter global.resolve_timeout from 5m to 10m.

   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   metadata:
     name: default
     namespace: service-telemetry
   spec:
     backends:
       metrics:
         prometheus:
           enabled: true
     alertmanagerConfigManifest: |
       apiVersion: v1
       kind: Secret
       metadata:
         name: 'alertmanager-default'
         namespace: 'service-telemetry'
       type: Opaque
       stringData:
         alertmanager.yaml: |-
           global:
             resolve_timeout: 10m
           route:
             group_by: ['job']
             group_wait: 30s
             group_interval: 5m
             repeat_interval: 12h
             receiver: 'null'
           receivers:
           - name: 'null'

   Copy to Clipboard Toggle word wrap

5. Verify that the configuration is applied to the secret:

   $ oc get secret alertmanager-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}'
   
   global:
     resolve_timeout: 10m
   route:
     group_by: ['job']
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: 'null'
   receivers:
   - name: 'null'

   Copy to Clipboard Toggle word wrap

6. To verify the configuration is loaded into Alertmanager, create a pod with access to curl:

   $ oc run curl --generator=run-pod/v1 --image=radial/busyboxplus:curl -i --tty

   Copy to Clipboard Toggle word wrap

7. Run curl against the alertmanager-operated service to retrieve the status and configYAML contents, and verify that the supplied configuration matches the configuration in Alertmanager:

   [ root@curl:/ ]$ curl alertmanager-operated:9093/api/v1/status
   
   {"status":"success","data":{"configYAML":"global:\n  resolve_timeout: 10m\n  http_config: {}\n  smtp_hello: localhost\n  smtp_require_tls: true\n  pagerduty_url: https://events.pagerduty.com/v2/enqueue\n  hipchat_api_url: https://api.hipchat.com/\n  opsgenie_api_url: https://api.opsgenie.com/\n  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/\n  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/\nroute:\n  receiver: \"null\"\n  group_by:\n  - job\n  group_wait: 30s\n  group_interval: 5m\n  repeat_interval: 12h\nreceivers:\n- name: \"null\"\ntemplates: []\n",...}}

   Copy to Clipboard Toggle word wrap
8. Verify that the configYAML field contains the changes you expect.

9. Exit the pod:

   [ root@curl:/ ]$ exit

   Copy to Clipboard Toggle word wrap

10. To clean up the environment, delete the curl pod:

    $ oc delete pod curl
    
    pod "curl" deleted

    Copy to Clipboard Toggle word wrap

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Use the oc command to edit the ServiceTelemetry object:

   $ oc edit stf default

   Copy to Clipboard Toggle word wrap

4. Add highAvailability.enabled: true to the spec section:

   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   ...
   spec:
     ...
     highAvailability:
       enabled: true

   Copy to Clipboard Toggle word wrap
5. Save your changes and close the object.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   $ oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Edit the ServiceTelemetry object:

   $ oc edit stf default

   Copy to Clipboard Toggle word wrap

4. Add the ...storage.strategy: ephemeral parameter to the spec section of the relevant component:

   apiVersion: infra.watch/v1beta1
   kind: ServiceTelemetry
   metadata:
     name: stf-default
     namespace: service-telemetry
   spec:
     alerting:
       enabled: true
       alertmanager:
         storage:
           strategy: ephemeral
     backends:
       metrics:
         prometheus:
           enabled: true
           storage:
             strategy: ephemeral
       events:
         elasticsearch:
           enabled: true
           storage:
             strategy: ephemeral

   Copy to Clipboard Toggle word wrap
5. Save your changes and close the object.