Chapter 4. Advanced features

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

3. Load the ServiceTelemetry object into an editor:

   oc edit servicetelemetry stf-default

4. To modify the ServiceTelemetry object, provide a manifest override parameter and the contents of the manifest to write to OCP instead of the defaults provided by STF.

   Note

   The trailing pipe (|) after entering the manifest override parameter indicates that the value provided is multi-line.

   $ oc edit servicetelemetry stf-default
   
   apiVersion: infra.watch/v1alpha1
   kind: ServiceTelemetry
   metadata:
     annotations:
       kubectl.kubernetes.io/last-applied-configuration: |
         {"apiVersion":"infra.watch/v1alpha1","kind":"ServiceTelemetry","metadata":{"annotations":{},"name":"stf-default","namespace":"service-telemetry"},"spec":{metricsEnabled":true}}
     creationTimestamp: "2020-04-14T20:29:42Z"
     generation: 1
     name: stf-default
     namespace: service-telemetry
     resourceVersion: "1949423"
     selfLink: /apis/infra.watch/v1alpha1/namespaces/service-telemetry/servicetelemetrys/stf-default
     uid: d058bc41-1bb0-49f5-9a8b-642f4b8adb95
   spec:
     metricsEnabled: true
     smartgatewayCollectdMetricsManifest: | 1
       apiVersion: smartgateway.infra.watch/v2alpha1
       kind: SmartGateway
       metadata:
         name: stf-default-collectd-telemetry
         namespace: service-telemetry
       spec:
         amqpUrl: stf-default-interconnect.service-telemetry.svc.cluster.local:5672/collectd/telemetry
         debug: true
         prefetch: 15000
         serviceType: metrics
         size: 1
         useTimestamp: true 2
   status:
     conditions:
     - ansibleResult:
         changed: 0
         completion: 2020-04-14T20:32:19.079508
         failures: 0
         ok: 52
         skipped: 1
       lastTransitionTime: "2020-04-14T20:29:59Z"
       message: Awaiting next reconciliation
       reason: Successful
       status: "True"
       type: Running

   1

   Manifest override parameter is defined in the spec of the ServiceTelemetry object.

   2

   End of the manifest override content.

5. Save and close.

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

3. Create a PrometheusRule object that contains the alert rule. The Prometheus Operator loads the rule into Prometheus:

   oc apply -f - <<EOF
   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     creationTimestamp: null
     labels:
       prometheus: stf-default
       role: alert-rules
     name: prometheus-alarm-rules
     namespace: service-telemetry
   spec:
     groups:
       - name: ./openstack.rules
         rules:
           - alert: Metric Listener down
             expr: collectd_qpid_router_status < 1 # To change the rule, edit the value of the expr parameter.
   EOF

4. To verify that the rules have been loaded into Prometheus by the Operator, create a pod with access to curl:

   oc run curl --generator=run-pod/v1 --image=radial/busyboxplus:curl -i --tty

5. Run curl to access the prometheus-operated service to return the rules loaded into memory:

   [ root@curl:/ ]$ curl prometheus-operated:9090/api/v1/rules
   {"status":"success","data":{"groups":[{"name":"./openstack.rules","file":"/etc/prometheus/rules/prometheus-stf-default-rulefiles-0/service-telemetry-prometheus-alarm-rules.yaml","rules":[{"name":"Metric Listener down","query":"collectd_qpid_router_status \u003c 1","duration":0,"labels":{},"annotations":{},"alerts":[],"health":"ok","type":"alerting"}],"interval":30}]}}

6. To verify that the output shows the rules loaded into the PrometheusRule object, for example the output contains the defined ./openstack.rules, exit from the pod:

   [ root@curl:/ ]$ exit

7. Clean up the environment by deleting the curl pod:

   $ oc delete pod curl
   
   pod "curl" deleted

1. Use the oc edit command:

   oc edit prometheusrules prometheus-alarm-rules

2. Edit the PrometheusRules manifest.
3. Save and close.

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

3. Edit the ServiceTelemetry object for your STF deployment

   oc edit servicetelemetry stf-default

4. Add a new parameter, alertmanagerConfigManifest and the Secret object contents to define the alertmanager.yaml configuration for Alertmanager:

   Note

   This loads the default template that is already managed by Service Telemetry Operator. To validate the changes are populating correctly, change a value, return the alertmanager-stf-default secret, and verify that the new value is loaded into memory, for example, changing the value global.resolve_timeout from 5m to 10m.

   apiVersion: infra.watch/v1alpha1
   kind: ServiceTelemetry
   metadata:
     name: stf-default
     namespace: service-telemetry
   spec:
     metricsEnabled: true
     alertmanagerConfigManifest: |
       apiVersion: v1
       kind: Secret
       metadata:
         name: 'alertmanager-stf-default'
         namespace: 'service-telemetry'
       type: Opaque
       stringData:
         alertmanager.yaml: |-
           global:
             resolve_timeout: 10m
           route:
             group_by: ['job']
             group_wait: 30s
             group_interval: 5m
             repeat_interval: 12h
             receiver: 'null'
           receivers:
           - name: 'null'

5. Verify that the configuration was applied to the secret:

   $ oc get secret alertmanager-stf-default -o go-template='{{index .data "alertmanager.yaml" | base64decode }}'
   
   global:
     resolve_timeout: 10m
   route:
     group_by: ['job']
     group_wait: 30s
     group_interval: 5m
     repeat_interval: 12h
     receiver: 'null'
   receivers:
   - name: 'null'

6. To verify the configuration has been loaded into Alertmanager, create a pod with access to curl:

   oc run curl --generator=run-pod/v1 --image=radial/busyboxplus:curl -i --tty

7. Run curl against the alertmanager-operated service to retrieve the status and configYAML contents and review the supplied configuration matches the configuration loaded into Alertmanager:

   [ root@curl:/ ]$ curl alertmanager-operated:9093/api/v1/status
   
   {"status":"success","data":{"configYAML":"global:\n  resolve_timeout: 10m\n  http_config: {}\n  smtp_hello: localhost\n  smtp_require_tls: true\n  pagerduty_url: https://events.pagerduty.com/v2/enqueue\n  hipchat_api_url: https://api.hipchat.com/\n  opsgenie_api_url: https://api.opsgenie.com/\n  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/\n  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/\nroute:\n  receiver: \"null\"\n  group_by:\n  - job\n  group_wait: 30s\n  group_interval: 5m\n  repeat_interval: 12h\nreceivers:\n- name: \"null\"\ntemplates: []\n",...}}

8. Verify that the configYAML field contains the expected changes. Exit from the pod:

   [ root@curl:/ ]$ exit

9. To clean up the environment, delete the curl pod:

   $ oc delete pod curl
   
   pod "curl" deleted

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

3. Use the oc command to edit the ServiceTelemetry object:

   $ oc edit ServiceTelemetry

4. Add highAvailabilityEnabled: true to the spec section:

   spec:
     eventsEnabled: true
     metricsEnabled: true
     highAvailabilityEnabled: true

5. Save your changes and close the object.

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

3. Clone the dashboard repository.

   git clone https://github.com/infrawatch/dashboards
   cd dashboards

4. Deploy the Grafana operator:

   oc create -f deploy/subscription.yaml

5. To verify that the operator launched successfully, run the oc get csv command. If the value of the PHASE column is Succeeded, the operator launched successfully:

   $ oc get csv
   
   NAME                                DISPLAY                                         VERSION   REPLACES                            PHASE
   grafana-operator.v3.2.0             Grafana Operator                                3.2.0                                         Succeeded
   ...

6. Launch a Grafana instance:

   $ oc create -f deploy/grafana.yaml

7. Verify that the Grafana instance deployed:

   $ oc get pod -l app=grafana
   
   NAME                                  READY   STATUS    RESTARTS   AGE
   grafana-deployment-7fc7848b56-sbkhv   1/1     Running   0          1m

8. Create the datasource and dashboard resources:

   oc create -f deploy/datasource.yaml \
       -f deploy/rhos-dashboard.yaml

9. Verify that the resources installed correctly:

   $ oc get grafanadashboards
   
   NAME             AGE
   rhos-dashboard   7d21h
   
   $ oc get grafanadatasources
   NAME                                  AGE
   service-telemetry-grafanadatasource   1m

10. Navigate to https://<grafana-route-address> in a web browser. Use the oc get routes command to retrieve the Grafana route address:

    oc get routes

11. To view the dashboard, click Dashboards and Manage.

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

3. Use the initially deployed Smart Gateways as a template for additional Smart Gateways. List the currently deployed Smart Gateways with the oc get smartgateways command. For example, if you deployed STF with metricsEnabled: true and eventsEnabled: true, the following Smart Gateways are displayed in the output:

   $ oc get smartgateways
   
   NAME                                         AGE
   stf-default-ceilometer-notification          14d
   stf-default-collectd-notification            14d
   stf-default-collectd-telemetry               14d

4. Retrieve the manifests for each Smart Gateway and store the contents in a temporary file, which you can modify later and use to create the new set of Smart Gateways:

   truncate --size 0 /tmp/cloud1-smartgateways.yaml && \
   for sg in $(oc get smartgateways -oname)
   do
     echo "---" >> /tmp/cloud1-smartgateways.yaml
     oc get ${sg} -oyaml --export >> /tmp/cloud1-smartgateways.yaml
   done

5. Modify the Smart Gateway manifest in the /tmp/cloud1-smartgateways.yaml file. Adjust the metadata.name and spec.amqpUrl fields to include the cloud identifier from your schema. For more information, see ]. To view example Smart Gateway manifests, see <<example-manifests_advanced-features[.

6. Deploy your new Smart Gateways:

   oc apply -f /tmp/cloud1-smartgateways.yaml

7. Verify that each Smart Gateway is running. This can take several minutes depending on the number of Smart Gateways:

   oc get po -l app=smart-gateway

1. Create the stf-connectors.yaml file and modify it to match the AMQP address that you want for this cloud deployment:

1. Ensure that the naming convention in the stf-connectors.yaml file aligns with the spec.amqpUrl field in the Smart Gateway configuration. For example, configure the CeilometerQdrEventsConfig.topic field to a value of cloud1-event.
2. Save the file in a directory for custom environment files, for example /home/stack/custom_templates/.

3. Source the authentication file:

   [stack@undercloud-0 ~]$ source stackrc
   
   (undercloud) [stack@undercloud-0 ~]$

4. Include the stf-connectors.yaml file in the overcloud deployment command, along with any other environment files relevant to your environment:

   (undercloud) [stack@undercloud-0 ~]$ openstack overcloud deploy \
   --templates /usr/share/openstack-tripleo-heat-templates \
   ...
   -e /home/stack/custom_templates/stf-connectors.yaml \
   ...

Procedure

1. Log in to Red Hat OpenShift Container Platform.

2. Change to the service-telemetry namespace:

   oc project service-telemetry

3. Edit the ServiceTelemetry object:

   $ oc edit ServiceTelemetry stf-default

4. Add the storageEphemeralEnabled: true parameter to the spec section:

   spec:
     eventsEnabled: true
     metricsEnabled: true
     storageEphemeralEnabled: true

5. Save your changes and close the object.