Chapter 6. Accessing Prometheus to monitor Fuse applications

Procedure

1. In a terminal window, add a service monitor to the namespace (project) in which you installed Prometheus:

   oc process -f  {$templates-base-url}/fuse-servicemonitor.yml -p NAMESPACE=<YOUR-NAMESPACE> -p FUSE_SERVICE_NAME=fuseonline | oc create -f -

2. In the OpenShift console, open the project and then select Applications Services.
3. Click the fuseonline service and then select Actions Edit YAML.
4. In the Editor, change the selector section of the YAML file by replacing app: fuseonline with syndesis.io/type: integration.
5. Click Save.

Procedure

1. Edit your Prometheus configuration file (prometheus-config.yml) as follows:

   1. Set the scrape interval to 5 seconds:

      global:
              scrape_interval:     5s
              evaluation_interval: 5s

   2. Add a scrape config job named integration-pod with the Kubernetes service discovery configuration that configures it to scrape pods in the ${OPENSHIFT_PROJECT}, which is typically the syndesis namespace:

       - job_name: integration-pods
      
                kubernetes_sd_configs:
                - role: pod
                  namespaces:
                    names:
                      - ${OPENSHIFT_PROJECT}

   3. Add a relabel_configs section that:

      • Only scrapes integration pods with the prometheus.io/scrape label set to true.
      • Sets the metrics_path and address labels, that are used to scrape the JMX exporter in the integration pod, by using values from the prometheus.io/path and prometheus.io/port labels.
      • Adds pod labels and annotations as Prometheus labels.

      • Creates kubernetes_namespace and kubernetes_pod_name labels.

        Here is an example relabel_configs section:

        relabel_configs:
                  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
                    action: keep
                    regex: true
                  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
                    action: replace
                    target_label: __metrics_path__
                    regex: (.+)
                  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
                    action: replace
                    regex: ([^:]+)(?::\d+)?;(\d+)
                    replacement: $1:$2
                    target_label: __address__
                  - action: labelmap
                    regex: __meta_kubernetes_pod_label_(.+)
                  - action: labelmap
                    regex: __meta_kubernetes_pod_annotation_(syndesis.+)
                  - source_labels: [__meta_kubernetes_namespace]
                    action: replace
                    target_label: kubernetes_namespace
                  - source_labels: [__meta_kubernetes_pod_name]
                    action: replace
                    target_label: kubernetes_pod_name

   4. Fuse Online integrations expose a large number of metrics from the JVM, Camel and CXF. To reduce the amount of storage needed for metrics, add the following metric_relabel_configs section to filter out metrics that are not displayed in the Fuse Online console.

       metric_relabel_configs:
                - source_labels: [__name__]
                  regex: jmx_(.+)
                  action: drop
                - source_labels: [__name__]
                  regex: jvm_(.+)
                  action: drop
                - source_labels: [__name__]
                  regex: process_(.+)
                  action: drop
                - source_labels: [type, __name__]
                  separator: ':'
                  regex: context:(org_apache_camel_ExchangesTotal|org_apache_camel_ExchangesFailed|io_syndesis_camel_StartTimestamp|io_syndesis_camel_LastExchangeCompletedTimestamp|io_syndesis_camel_LastExchangeFailureTimestamp)
                  action: keep

      Note

      The last configuration line explicitly lists metrics to be added in Prometheus metrics store that are critical to the statistics shown in the Fuse Online web console. Your Prometheus instance must explicitly allow these metrics to be collected if other metrics are being filtered.

2. Update the Prometheus pod configuration to store 30 days worth of metric data as follows:

   args:
               - '--config.file=/etc/prometheus/prometheus.yml'
               - '--storage.tsdb.retention=30d'