Este contenido no está disponible en el idioma seleccionado.
Chapter 3. Enabling 3scale API Management monitoring
You can enable the Red Hat 3scale API Management monitoring using Prometheus and Grafana operators. Prometheus is an open-source monitoring and alerting tool designed to collect and analyze metrics from applications and infrastructure in real time. Grafana is an open-source analytics and visualization platform used to display and monitor real-time data from various sources in customizable dashboards.
With the deprecation of Grafana 4 on OpenShift 4.16 and later, the instructions are split into two sections:
- For OpenShift 4.15 and earlier.
- For OpenShift 4.16 and later.
- Grafana 5 is available on OpenShift 4.14 or later and supported by the 3scale 2.15 operator; using Grafana 5 is recommended when possible.
- Red Hat support for Prometheus and Grafana is limited to the configuration recommendations provided in Red Hat product documentation.
- The 3scale operator creates monitoring resources, but does not prevent modification of those resources.
- You must install the 3scale operator and Prometheus operator in the same namespace or use cluster-wide operators.
The steps for migrating from Grafana 4 to Grafana 5 are also covered.
Prerequisites
- The 3scale operator is installed.
The Prometheus operator is installed from the OperatorHub. You can use the Prometheus operator for creating and managing Prometheus instances. It provides the
Prometheuscustom resource definition (CRD) required by 3scale monitoring.The following Prometheus operator version is compatible with 3scale depending on your version of OpenShift Container Platform (OCP):
- Latest Prometheus community operator with OCP 4.15 or earlier.
- Latest Prometheus community operator with OCP 4.16 or later.
The Grafana operator is installed from the OperatorHub. You can use the Grafana operator for creating and managing Grafana instances. It provides the
GrafanaDashboardCRD required by 3scale monitoring.The following Grafana operator versions are compatible with 3scale depending on your version of OCP:
- Grafana community operator 4 with OCP 4.15 or earlier.
- Grafana community operator 5 with OCP 4.16 or later.
If your cluster is exposed on the Internet, make sure to protect the Prometheus and Grafana services.
This section describes how to enable monitoring of a 3scale instance, so that you can view the Grafana dashboards.
3.1. Configuring Grafana 4 and Prometheus to OCP 4.15 Copiar enlaceEnlace copiado en el portapapeles!
Configure Grafana and Prometheus for OpenShift Container Platform (OCP) versions up to 4.15. This guide walks through setting up these tools to monitor your 3scale environment, ensuring you get real-time insights and data visualization.
Procedure
Install Grafana 4 community operator from the OperatorHub.
- Log in to the OpenShift Container Platform (OCP) using your OpenShift administrator credentials.
Select the project from the Project list where you want to install the Grafana community operator.
ImportantInstall the Grafana community operator in the same project where you installed the 3scale operator.
- Navigate to Operators > OperatorHub.
- Search for "grafana" and click Grafana Operator.
Click Install on the Grafana Community Operator page. The Create Operator Subscription page is shown. Complete the following steps to create the Grafana operator subscription:
- Click A specific namespace on the cluster and choose the project where you want to install the Grafana community operator.
- Click Subscribe.
- Click Approve.
Install latest Prometheus community operator from the OperatorHub.
- Log in to the OpenShift Container Platform (OCP) using your OpenShift administrator credentials.
Select the project from the Project list where you want to install the Prometheus community operator.
ImportantInstall the Prometheus community operator in the same project where you installed the 3scale operator.
- Navigate to Operators > OperatorHub.
- Search for "prometheus" and click Prometheus Operator.
Click Install on the Prometheus Community Operator page. The Create Operator Subscription page is shown. Complete the following steps to create the Prometheus operator subscription:
- Click A specific namespace on the cluster and choose the project where you want to install the Prometheus operator.
- Click Subscribe.
- Click Approve.
Enable monitoring by setting the
spec.monitoring.enabledparameter of your 3scale deployment YAML totrue. For example:apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: ... spec: monitoring: enabled: true ...Log in to your OpenShift cluster. You must log in as a user with an edit cluster role in the OpenShift project of the 3scale, for example,
cluster-admin.$ oc loginSwitch to your 3scale project.
$ oc project <project_name>
Create a new service account for Prometheus:
$ oc create serviceaccount prometheus-monitoringCreate a ClusterRoleBinding to give Prometheus ServiceAccount the Role-based access control (RBAC) permissions required to scrape metrics. Update the ServiceAccount namespace before creating the ClusterRoleBinding:
$ oc adm policy add-cluster-role-to-user cluster-monitoring-view -z prometheus-monitoring -n "<3scale_namespace>Create a token for the
prometheus-monitoringServiceAccount:$ oc create token prometheus-monitoringPrometheus will lose access to the required resource when this token expires. Add the
--duration X[s|m|h]to specify how long the token is valid for.-
Update the file 3scale-scrape-configs.yaml
bearer_tokenfield with the token generated you generated. Create additional-scrape-config secret:
$ oc create secret generic additional-scrape-configs --from-file=3scale-scrape-configs.yaml=./3scale-scrape-configs.yamlDeploy Prometheus.
In Prometheus.yaml file, fill the
spec.externalUrlfield with the external URL. The URL template should be:spec: ... externalUrl: https://prometheus.<namespace-name>.apps.<cluster-domain>
Deploy Prometheus server:
$ oc apply -f prometheus.yamlCreate the Prometheus route:
$ oc expose service prometheus-operated --hostname prometheus.<namespace-name>.apps.<cluster-name>Deploy Grafana datasource:
$ oc apply -f datasource-v4.yamlDeploy Grafana
$ oc apply -f grafana-v4.yaml
Additional resources
3.2. Configuring Grafana and Prometheus for OCP 4.16 Copiar enlaceEnlace copiado en el portapapeles!
Configure Grafana and Prometheus for OpenShift Container Platform (OCP) 4.16. This guide covers the steps to set up monitoring and data visualization, optimized for OCP 4.16.
Procedure
Install Grafana 5 community operator from the OperatorHub.
- Log in to the OpenShift Container Platform (OCP) using your OpenShift administrator credentials.
- Select the project from the Project list where you want to install the Grafana community operator.
- Navigate to Operators > OperatorHub.
- Search for "grafana" and click Grafana Operator.
Click Install on the Grafana Community Operator page. The Create Operator Subscription page is shown. Complete the following steps to create the Grafana operator subscription:
- Click A specific namespace on the cluster and choose the project where you want to install the Grafana community operator.
- Click Subscribe.
- Click Approve.
Install the latest Prometheus community operator from the OperatorHub.
- Log in to the OpenShift Container Platform (OCP) using your OpenShift administrator credentials.
Select the project from the Project list where you want to install the Prometheus community operator.
ImportantInstall the Prometheus community operator in the same project where you installed the 3scale operator.
- Navigate to Operators > OperatorHub.
- Search for "prometheus" and click Prometheus Operator.
Click Install on the Prometheus Community Operator page. The Create Operator Subscription page is shown. Complete the following steps to create the Prometheus operator subscription:
- Click A specific namespace on the cluster and choose the project where you want to install the Prometheus community operator.
- Click Subscribe.
- Click Approve.
Enable monitoring by setting the
spec.monitoring.enabledparameter of your 3scale deployment YAML totrue. For example:apiVersion: apps.3scale.net/v1alpha1 kind: APIManager metadata: ... spec: monitoring: enabled: true ...Log in to your OpenShift cluster. You must log in as a user with an edit cluster role in the OpenShift project of the 3scale, for example,
cluster-admin.$ oc loginSwitch to your 3scale project.
$ oc project <project_name>
Create a new service account for Prometheus:
$ oc create serviceaccount prometheus-monitoringCreate a ClusterRoleBinding to give Prometheus ServiceAccount the Role-based access control (RBAC) permissions required to scrape metrics. Update the ServiceAccount namespace before creating the ClusterRoleBinding:
$ oc adm policy add-cluster-role-to-user cluster-monitoring-view -z prometheus-monitoring -n "<3scale_namespace>"Create a token for the
prometheus-monitoringServiceAccount:$ oc create token prometheus-monitoringPrometheus will lose access to the required resource when this token expires. Add the
--duration X[s|m|h]to specify how long the token is valid for.-
Update the file 3scale-scrape-configs.yaml
bearer_tokenfield with the token generated you generated. Create additional-scrape-config secret:
$ oc create secret generic additional-scrape-configs --from-file=3scale-scrape-configs.yaml=./3scale-scrape-configs.yamlDeploy Prometheus.
In Prometheus.yaml file, fill the
spec.externalUrlfield with the external URL. The URL template should be:spec: ... externalUrl: https://prometheus.<namespace-name>.apps.<cluster-domain>
Deploy Prometheus server:
$ oc apply -f prometheus.yamlCreate the Prometheus route:
$ oc expose service prometheus-operated --hostname prometheus.<namespace-name>.apps.<cluster-name>Deploy Grafana datasource:
$ oc apply -f datasource-v5.yamlDeploy Grafana:
$ oc apply -f grafana-v5.yamlExpose grafana route:
$ oc expose service example-grafana-service
Additional resources
3.3. Migrating Grafana 4 to Grafana 5 on OpenShift Container Platform 4.16 Copiar enlaceEnlace copiado en el portapapeles!
This step-by-step guide explains how to migrate from Grafana 4 to Grafana 5 on OpenShift Container Platform (OCP) 4.16 or later. The process covers removing Grafana 4, setting up Grafana 5, and ensuring all necessary components are updated.
Make sure to back up any important data and dashboards before starting the migration.
3.3.1. Removing Grafana 4 Copiar enlaceEnlace copiado en el portapapeles!
Procedure
Remove the Grafana 4 custom resource (CR):
ImportantRemoving the Grafana 4 CR deletes the Grafana 4 application along with the service and routes.
$ oc delete grafana <grafana-cr-name> -n <namespace>Verify route removal:
$ oc get route -n <namespace>Delete the route manually if it still exists:
$ oc delete route <grafana-route-name> -n <namespace>
Remove the Grafana 4 datasource CR:
$ oc delete grafanadatasource <datasource-cr-name> -n <namespace>Remove the Grafana 4 operator:
$ oc delete subscription <grafana-operator-subscription> -n <namespace> $ oc delete clusterserviceversion <grafana-operator-csv-name> -n <namespace>
3.3.2. Installing Grafana 5 Copiar enlaceEnlace copiado en el portapapeles!
Procedure
Install Grafana 5:
Follow the procedure to Configure Grafana and Prometheus for OCP 4.16
Restart the Prometheus instance to apply the new configuration:
$ oc rollout restart statefulset prometheus-k8s -n <namespace>
3.3.3. Migrating custom dashboards to Grafana 5 CRDs Copiar enlaceEnlace copiado en el portapapeles!
Procedure
Migrate your custom dashboards to Grafana 5 CRDs.
Convert the exported JSON dashboards to Custom Resource Definitions (CRDs) for Grafana v5. Example YAML for a GrafanaDashboard CR:
apiVersion: integreatly.org/v1alpha1 kind: GrafanaDashboard metadata: name: example-dashboard namespace: <namespace> spec: json: | { "dashboard": { ... } }Apply the dashboard CRD:
$ oc apply -f grafana-dashboard.yaml
3.3.4. Optional: Removing Grafana 4 CRDs Copiar enlaceEnlace copiado en el portapapeles!
Procedure
Delete Grafana 4 dashboards CRDs:
$ oc delete crd grafanadashboards.integreatly.orgImportantIf you choose not to remove the CRDs, the 3scale operator continues to reconcile Grafana 4 dashboards, but they do not have any impact on the monitoring stack.
3.3.5. Restarting Grafana operator and Grafana deployment Copiar enlaceEnlace copiado en el portapapeles!
Procedure
Restart the 3scale operator:
$ oc rollout restart deployment/3scale-operator -n <namespace>Restart Grafana operator and Grafana deployment:
$ oc rollout restart deployment/grafana-operator -n <namespace> $ oc rollout restart deployment/<grafana-deployment-name> -n <namespace>
3.4. Viewing metrics for 3scale API Management Copiar enlaceEnlace copiado en el portapapeles!
After configuring 3scale, Prometheus, and Grafana you can view the metrics described in this section.
Procedure
- Log into the Grafana console.
Check that you can view metrics for the following:
- Kubernetes resources at pod and namespace level where 3scale is installed
- APIcast Staging
- APIcast Production
- Backend worker
- Backend listener
- System
- Zync
3.5. 3scale API Management system metrics exposed to Prometheus Copiar enlaceEnlace copiado en el portapapeles!
You can configure the following ports to use 3scale system pods with Prometheus endpoints to expose metrics.
| system-app | Port |
|---|---|
|
| 9394 |
|
| 9395 |
|
| 9396 |
| system-sidekiq | Port |
|---|---|
|
| 9394 |
The endpoints are only accessible internally using:
http://${service}:${port}/metrics
For example:
http://system-developer:9394/metrics