Chapter 4. Advanced features

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Load the ServiceTelemetry object into an editor:

   oc edit servicetelemetry stf-default

   Copy to Clipboard Toggle word wrap

4. To modify the ServiceTelemetry object, provide a manifest override parameter and the contents of the manifest to write to OCP instead of the defaults provided by STF.

   Note

   The trailing pipe (|) after entering the manifest override parameter indicates that the value provided is multi-line.

   $ oc edit servicetelemetry stf-default
   
   apiVersion: infra.watch/v1alpha1
   kind: ServiceTelemetry
   metadata:
     annotations:
       kubectl.kubernetes.io/last-applied-configuration: |
         {"apiVersion":"infra.watch/v1alpha1","kind":"ServiceTelemetry","metadata":{"annotations":{},"name":"stf-default","namespace":"service-telemetry"},"spec":{metricsEnabled":true}}
     creationTimestamp: "2020-04-14T20:29:42Z"
     generation: 1
     name: stf-default
     namespace: service-telemetry
     resourceVersion: "1949423"
     selfLink: /apis/infra.watch/v1alpha1/namespaces/service-telemetry/servicetelemetrys/stf-default
     uid: d058bc41-1bb0-49f5-9a8b-642f4b8adb95
   spec:
     metricsEnabled: true
     smartgatewayCollectdMetricsManifest: | 

   1

   
       apiVersion: smartgateway.infra.watch/v2alpha1
       kind: SmartGateway
       metadata:
         name: stf-default-collectd-telemetry
         namespace: service-telemetry
       spec:
         amqpUrl: stf-default-interconnect.service-telemetry.svc.cluster.local:5672/collectd/telemetry
         debug: true
         prefetch: 15000
         serviceType: metrics
         size: 1
         useTimestamp: true 

   2

   
   status:
     conditions:
     - ansibleResult:
         changed: 0
         completion: 2020-04-14T20:32:19.079508
         failures: 0
         ok: 52
         skipped: 1
       lastTransitionTime: "2020-04-14T20:29:59Z"
       message: Awaiting next reconciliation
       reason: Successful
       status: "True"
       type: Running

   Copy to Clipboard Toggle word wrap

   1

   Manifest override parameter is defined in the spec of the ServiceTelemetry object.

   2

   End of the manifest override content.

5. Save and close.

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Create a PrometheusRule object that contains the alert rule. The Prometheus Operator loads the rule into Prometheus:

   oc apply -f - <<EOF
   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     creationTimestamp: null
     labels:
       prometheus: stf-default
       role: alert-rules
     name: prometheus-alarm-rules
     namespace: service-telemetry
   spec:
     groups:
       - name: ./openstack.rules
         rules:
           - alert: Metric Listener down
             expr: collectd_qpid_router_status < 1 # To change the rule, edit the value of the expr parameter.
   EOF

   Copy to Clipboard Toggle word wrap

4. To verify that the rules have been loaded into Prometheus by the Operator, create a pod with access to curl:

   oc run curl --generator=run-pod/v1 --image=radial/busyboxplus:curl -i --tty

   Copy to Clipboard Toggle word wrap

5. Run curl to access the prometheus-operated service to return the rules loaded into memory:

   [ root@curl:/ ]$ curl prometheus-operated:9090/api/v1/rules
   {"status":"success","data":{"groups":[{"name":"./openstack.rules","file":"/etc/prometheus/rules/prometheus-stf-default-rulefiles-0/service-telemetry-prometheus-alarm-rules.yaml","rules":[{"name":"Metric Listener down","query":"collectd_qpid_router_status \u003c 1","duration":0,"labels":{},"annotations":{},"alerts":[],"health":"ok","type":"alerting"}],"interval":30}]}}

   Copy to Clipboard Toggle word wrap

6. To verify that the output shows the rules loaded into the PrometheusRule object, for example the output contains the defined ./openstack.rules, exit from the pod:

   [ root@curl:/ ]$ exit

   Copy to Clipboard Toggle word wrap

7. Clean up the environment by deleting the curl pod:

   $ oc delete pod curl
   
   pod "curl" deleted

   Copy to Clipboard Toggle word wrap

1. Use the oc edit command:

   oc edit prometheusrules prometheus-alarm-rules

   Copy to Clipboard Toggle word wrap
2. Edit the PrometheusRules manifest.
3. Save and close.

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Edit the ServiceTelemetry object for your STF deployment

   oc edit servicetelemetry stf-default

   Copy to Clipboard Toggle word wrap

4. Add a new parameter, alertmanagerConfigManifest and the Secret object contents to define the alertmanager.yaml configuration for Alertmanager:

   Note

   This loads the default template that is already managed by Service Telemetry Operator. To validate the changes are populating correctly, change a value, return the alertmanager-stf-default secret, and verify that the new value is loaded into memory, for example, changing the value global.resolve_timeout from 5m to 10m.

   apiVersion: infra.watch/v1alpha1
   kind: ServiceTelemetry
   metadata:
     name: stf-default
     namespace: service-telemetry
   spec:
     metricsEnabled: true
     alertmanagerConfigManifest: |
       apiVersion: v1
       kind: Secret
       metadata:
         name: 'alertmanager-stf-default'
         namespace: 'service-telemetry'
       type: Opaque
       stringData:
         alertmanager.yaml: |-
           global:
             resolve_timeout: 10m
           route:
             group_by: ['job']
             group_wait: 30s
             group_interval: 5m
             repeat_interval: 12h
             receiver: 'null'
           receivers:
           - name: 'null'

   Copy to Clipboard Toggle word wrap

5. Verify that the configuration was applied to the secret:

   $ oc get secret alertmanager-stf-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}'
   
   global:
     resolve_timeout: 10m
   route:
     group_by: ['job']
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: 'null'
   receivers:
   - name: 'null'

   Copy to Clipboard Toggle word wrap

6. To verify the configuration has been loaded into Alertmanager, create a pod with access to curl:

   oc run curl --generator=run-pod/v1 --image=radial/busyboxplus:curl -i --tty

   Copy to Clipboard Toggle word wrap

7. Run curl against the alertmanager-operated service to retrieve the status and configYAML contents and review the supplied configuration matches the configuration loaded into Alertmanager:

   [ root@curl:/ ]$ curl alertmanager-operated:9093/api/v1/status
   
   {"status":"success","data":{"configYAML":"global:\n  resolve_timeout: 10m\n  http_config: {}\n  smtp_hello: localhost\n  smtp_require_tls: true\n  pagerduty_url: https://events.pagerduty.com/v2/enqueue\n  hipchat_api_url: https://api.hipchat.com/\n  opsgenie_api_url: https://api.opsgenie.com/\n  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/\n  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/\nroute:\n  receiver: \"null\"\n  group_by:\n  - job\n  group_wait: 30s\n  group_interval: 5m\n  repeat_interval: 12h\nreceivers:\n- name: \"null\"\ntemplates: []\n",...}}

   Copy to Clipboard Toggle word wrap

8. Verify that the configYAML field contains the expected changes. Exit from the pod:

   [ root@curl:/ ]$ exit

   Copy to Clipboard Toggle word wrap

9. To clean up the environment, delete the curl pod:

   $ oc delete pod curl
   
   pod "curl" deleted

   Copy to Clipboard Toggle word wrap

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Use the oc command to edit the ServiceTelemetry object:

   $ oc edit ServiceTelemetry

   Copy to Clipboard Toggle word wrap

4. Add highAvailabilityEnabled: true to the spec section:

   spec:
     eventsEnabled: true
     metricsEnabled: true
     highAvailabilityEnabled: true

   Copy to Clipboard Toggle word wrap
5. Save your changes and close the object.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Clone the dashboard repository.

   git clone https://github.com/infrawatch/dashboards
   cd dashboards

   Copy to Clipboard Toggle word wrap

4. Deploy the Grafana operator:

   oc create -f deploy/subscription.yaml

   Copy to Clipboard Toggle word wrap

5. To verify that the operator launched successfully, run the oc get csv command. If the value of the PHASE column is Succeeded, the operator launched successfully:

   $ oc get csv
   
   NAME                                DISPLAY                                         VERSION   REPLACES                            PHASE
   grafana-operator.v3.2.0             Grafana Operator                                3.2.0                                         Succeeded
   ...

   Copy to Clipboard Toggle word wrap

6. Launch a Grafana instance:

   $ oc create -f deploy/grafana.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that the Grafana instance deployed:

   $ oc get pod -l app=grafana
   
   NAME                                  READY   STATUS    RESTARTS   AGE
   grafana-deployment-7fc7848b56-sbkhv   1/1     Running   0          1m

   Copy to Clipboard Toggle word wrap

8. Create the datasource and dashboard resources:

   oc create -f deploy/datasource.yaml \
       -f deploy/rhos-dashboard.yaml

   Copy to Clipboard Toggle word wrap

9. Verify that the resources installed correctly:

   $ oc get grafanadashboards
   
   NAME             AGE
   rhos-dashboard   7d21h
   
   $ oc get grafanadatasources
   NAME                                  AGE
   service-telemetry-grafanadatasource   1m

   Copy to Clipboard Toggle word wrap

10. Navigate to https://<grafana-route-address> in a web browser. Use the oc get routes command to retrieve the Grafana route address:

    oc get routes

    Copy to Clipboard Toggle word wrap
11. To view the dashboard, click Dashboards and Manage.

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Use the initially deployed Smart Gateways as a template for additional Smart Gateways. List the currently deployed Smart Gateways with the oc get smartgateways command. For example, if you deployed STF with metricsEnabled: true and eventsEnabled: true, the following Smart Gateways are displayed in the output:

   $ oc get smartgateways
   
   NAME                                         AGE
   stf-default-ceilometer-notification          14d
   stf-default-collectd-notification            14d
   stf-default-collectd-telemetry               14d

   Copy to Clipboard Toggle word wrap

4. Retrieve the manifests for each Smart Gateway and store the contents in a temporary file, which you can modify later and use to create the new set of Smart Gateways:

   truncate --size 0 /tmp/cloud1-smartgateways.yaml && \
   for sg in $(oc get smartgateways -oname)
   do
     echo "---" >> /tmp/cloud1-smartgateways.yaml
     oc get ${sg} -oyaml --export >> /tmp/cloud1-smartgateways.yaml
   done

   Copy to Clipboard Toggle word wrap
5. Modify the Smart Gateway manifest in the /tmp/cloud1-smartgateways.yaml file. Adjust the metadata.name and spec.amqpUrl fields to include the cloud identifier from your schema. For more information, see ]. To view example Smart Gateway manifests, see <<example-manifests_advanced-features[.

6. Deploy your new Smart Gateways:

   oc apply -f /tmp/cloud1-smartgateways.yaml

   Copy to Clipboard Toggle word wrap

7. Verify that each Smart Gateway is running. This can take several minutes depending on the number of Smart Gateways:

   oc get po -l app=smart-gateway

   Copy to Clipboard Toggle word wrap

1. Create the stf-connectors.yaml file and modify it to match the AMQP address that you want for this cloud deployment:

1. Ensure that the naming convention in the stf-connectors.yaml file aligns with the spec.amqpUrl field in the Smart Gateway configuration. For example, configure the CeilometerQdrEventsConfig.topic field to a value of cloud1-event.
2. Save the file in a directory for custom environment files, for example /home/stack/custom_templates/.

3. Source the authentication file:

   [stack@undercloud-0 ~]$ source stackrc
   
   (undercloud) [stack@undercloud-0 ~]$

   Copy to Clipboard Toggle word wrap

4. Include the stf-connectors.yaml file in the overcloud deployment command, along with any other environment files relevant to your environment:

   (undercloud) [stack@undercloud-0 ~]$ openstack overcloud deploy \
   --templates /usr/share/openstack-tripleo-heat-templates \
   ...
   -e /home/stack/custom_templates/stf-connectors.yaml \
   ...

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

   Copy to Clipboard Toggle word wrap

3. Edit the ServiceTelemetry object:

   $ oc edit ServiceTelemetry stf-default

   Copy to Clipboard Toggle word wrap

4. Add the storageEphemeralEnabled: true parameter to the spec section:

   spec:
     eventsEnabled: true
     metricsEnabled: true
     storageEphemeralEnabled: true

   Copy to Clipboard Toggle word wrap
5. Save your changes and close the object.