Chapter 3. Monitoring 3scale

Procedure

1. Configure 3scale to enable monitoring by setting the spec.monitoring.enabled parameter of the 3scale deployment YAML to true. For example:

   1. Create an APIManager custom resource named 3scale-monitoring.yml to enable monitoring:

      apiVersion: apps.3scale.net/v1alpha1
      kind: APIManager
      metadata:
        name: apimanager1
      spec:
        wildcardDomain: example.com
        monitoring:
          enabled: true
          enablePrometheusRules: false 

      1

      Copy to Clipboard Toggle word wrap

      1

      You can optionally disable PrometheusRules, which is otherwise enabled by default.

   2. Log in to your OpenShift cluster. You must log in as a user with an edit cluster role in the OpenShift project of the 3scale, for example, cluster-admin.

      oc login

      Copy to Clipboard Toggle word wrap

   3. Switch to your 3scale project.

      oc project <project_name>

      Copy to Clipboard Toggle word wrap

   4. Deploy the custom resource:

      $ oc apply -f 3scale-monitoring.yml

      Copy to Clipboard Toggle word wrap

Procedure

1. Deploy the Prometheus custom resource as follows depending on whether you want to monitor all resources in the cluster or only 3scale resources:

   • To monitor all resources in the cluster, set the spec.podMonitorSelector attribute to {} and set the spec.ruleSelector attribute to {}. For example, apply the following custom resource:

     apiVersion: monitoring.coreos.com/v1
     kind: Prometheus
     metadata:
       name: example
     spec:
       podMonitorSelector: {}
       ruleSelector: {}

     Copy to Clipboard Toggle word wrap

   • If you deployed 3scale and the Prometheus operator in the same OpenShift project, and assuming the value of APP_LABEL is set to the default 3scale-api-management, monitor 3scale resources using the following steps:

     1. Set the spec.podMonitorSelector attribute to:

         podMonitorSelector:
          matchExpressions:
          - key: app
              operator: In
              values:
              - 3scale-api-management

        Copy to Clipboard Toggle word wrap

     2. Set the spec.ruleSelector attribute to:

           matchExpressions:
           - key: app
             operator: In
             values:
             - 3scale-api-management

        Copy to Clipboard Toggle word wrap

        For example, apply the following custom resource:

        apiVersion: monitoring.coreos.com/v1
        kind: Prometheus
        metadata:
          name: example
        spec:
         podMonitorSelector:
          matchExpressions:
          - key: app
              operator: In
              values:
              - 3scale-api-management
         ruleSelector:
           matchExpressions:
           - key: app
             operator: In
             values:
             - 3scale-api-management

        Copy to Clipboard Toggle word wrap

   • If you deployed 3scale and the Prometheus operator in different OpenShift projects, monitor 3scale resources using the following steps:

     1. Label the OpenShift project where 3scale is deployed with MYLABELKEY=MYLABELVALUE

     2. Use a podMonitorNamespaceSelector filter to select the 3scale pods. For example, apply the following custom resource:

        apiVersion: monitoring.coreos.com/v1
        kind: Prometheus
        metadata:
          name: example
        spec:
         podMonitorSelector: {}
         ruleSelector: {}
         podMonitorNamespaceSelector:
           matchExpressions:
           - key: MYLABELKEY
             operator: In
             values:
             - MYLABELVALUE

        Copy to Clipboard Toggle word wrap

2. To ensure that dashboards and alerts work as expected, you must incorporate Kubernetes metrics, that is, kube-state-metrics, by performing one of the following:

   • Federate the Prometheus instance with the cluster default Prometheus instance.
   • Configure your own scraping jobs to get metrics from kubelet, etcd and others.

Procedure

1. Make sure Grafana services are configured to monitor the GrafanaDashboards resources by overwriting the app=3scale-api-management label. For example, apply the following custom resource:

   apiVersion: integreatly.org/v1alpha1
   kind: Grafana
   metadata:
     name: grafana
   spec:
     dashboardLabelSelector:
     - matchExpressions:
       - key: app
         operator: In
         values:
         - 3scale-api-management

   Copy to Clipboard Toggle word wrap

   Grafana Dashboards created by the 3scale operator are labeled as follows:

   app: 3scale-api-management
   monitoring-key: middleware

   Copy to Clipboard Toggle word wrap
2. If the Grafana operator is installed in a different namespace than 3scale, configure it to monitor resources outside the namespace using the --namespaces or --scan-all operator flags. See the Grafana documentation for more information about the operator flags.

3. Create a GrafanaDataSource custom resource of type prometheus to make the Prometheus data available in Grafana. For example:

   apiVersion: integreatly.org/v1alpha1
   kind: GrafanaDataSource
   metadata:
     name: prometheus
   spec:
     name: middleware
     datasources:
       - name: Prometheus
         type: prometheus
         access: proxy
         url: http://prometheus-operated:9090
         isDefault: true
         version: 1
         editable: true
         jsonData:
           timeInterval: "5s"

   Copy to Clipboard Toggle word wrap

   where http://prometheus-operated:9090 is the Prometheus route.

4. Make sure permissions are set correctly as described in the Grafana documentation.

Procedure

1. Log into the Grafana console.

2. Check that you can view metrics for the following:

   • Kubernetes resources at pod and namespace level where 3scale is installed
   • APIcast Staging
   • APIcast Production
   • Backend worker
   • Backend listener
   • System
   • Zync