Chapter 2. Monitoring
2.1. Monitoring with GitOps dashboards
You can access a graphical view of GitOps instances with Red Hat OpenShift GitOps monitoring dashboards to observe the behavior and usage of each instance across the cluster.
There are three GitOps dashboards available:
- GitOps Overview: See an overview of all GitOps instances installed on the cluster, including the number of applications, health and sync status, application and sync activity.
- GitOps Components: View detailed information, such as CPU or memory, for application-controller, repo-server, server, and other GitOps components.
- GitOps gRPC Services: View metrics related to gRPC service activity between the various components in Red Hat OpenShift GitOps.
2.1.1. Accessing GitOps monitoring dashboards
The monitoring dashboards are deployed automatically by the Operator. You can access GitOps monitoring dashboards from the Administrator perspective of the OpenShift Container Platform web console.
Disabling or changing the content of the dashboards is not supported.
Prerequisites
- You have access to the OpenShift Container Platform web console.
-
The Red Hat OpenShift GitOps Operator is installed in the default namespace,
openshift-gitops-operator
. -
The cluster monitoring is enabled on the
openshift-gitops-operator
namespace. -
You have installed an Argo CD application in your defined namespace, for example,
openshift-gitops
.
Procedure
-
In the Administrator perspective of the web console, go to Observe
Dashboards. - From the Dashboard drop-down list, select the desired GitOps dashboard: GitOps (Overview), GitOps / Components, or GitOps / gRPC Services.
- Optional: Choose a specific namespace, cluster, and interval from the Namespace, Cluster, and Interval drop-down lists.
- View the desired GitOps metrics in the GitOps dashboard.
2.2. Monitoring Argo CD instances
By default, the Red Hat OpenShift GitOps Operator automatically detects an installed Argo CD instance in your defined namespace, for example, openshift-gitops
, and connects it to the monitoring stack of the cluster to provide alerts for out-of-sync applications.
2.2.1. Prerequisites
-
You have access to the cluster with
cluster-admin
privileges. - You have access to the OpenShift Container Platform web console.
- You have installed the Red Hat OpenShift GitOps Operator in your cluster.
-
You have installed an Argo CD application in your defined namespace, for example,
openshift-gitops
.
2.2.2. Monitoring Argo CD health using Prometheus metrics
You can monitor the health status of an Argo CD application by running Prometheus metrics queries against it.
Procedure
-
In the Developer perspective of the web console, select the namespace where your Argo CD application is installed, and navigate to Observe
Metrics. - From the Select query drop-down list, select Custom query.
To check the health status of your Argo CD application, enter the Prometheus Query Language (PromQL) query similar to the following example in the Expression field:
Example
sum(argocd_app_info{dest_namespace=~"<your_defined_namespace>",health_status!=""}) by (health_status) 1
- 1
- Replace the
<your_defined_namespace>
variable with the actual name of your defined namespace, for exampleopenshift-gitops
.
2.3. Monitoring the GitOps Operator performance
The Red Hat OpenShift GitOps Operator emits metrics about its performance. With the OpenShift monitoring stack that picks up these metrics, you can monitor and analyze the Operator’s performance. The Operator exposes the following metrics, which you can view by using the OpenShift Container Platform web console:
Metric name | Type | Description |
---|---|---|
| Gauge | The total number of active Argo CD instances currently managed by the Operator across the cluster at a given time. |
| Gauge | The number of active Argo CD instances in a given phase, such as pending, or available. |
| Counter | The total number of reconciliations that have occurred for an instance in a given namespace at any given time. |
| Counter |
The number of reconciliation cycles completed under given time durations for an instance. For example, |
| Counter | The total number of reconciliation cycles observed for a given instance. |
| Counter | The total amount of time taken for the observed reconciliations for a given instance. |
Gauge is a value that can go up or down. Counter is a value that can only go up.
2.3.1. Accessing the GitOps Operator metrics
You can access the Operator metrics from the Administrator perspective of the OpenShift Container Platform web console to track the performance of the Operator.
Prerequisites
- You have access to the OpenShift Container Platform web console.
-
The Red Hat OpenShift GitOps Operator is installed in the default
openshift-gitops-operator
namespace. -
The cluster monitoring is enabled on the
openshift-gitops-operator
namespace.
Procedure
-
In the Administrator perspective of the web console, go to Observe
Metrics. Enter the metric in the Expression field. You can choose from the following metrics:
-
active_argocd_instances_total
-
active_argocd_instances_by_phase
-
active_argocd_instance_reconciliation_count
-
controller_runtime_reconcile_time_seconds_per_instance_bucket
-
controller_runtime_reconcile_time_seconds_per_instance_count
-
controller_runtime_reconcile_time_seconds_per_instance_sum
-
(Optional): Filter the metric by its properties. For example, filter the
active_argocd_instances_by_phase
metric by theAvailable
phase:Example
active_argocd_instances_by_phase{phase="Available"}
- (Optional): Click Add query to enter multiple queries.
- Click Run queries to enable and observe the GitOps Operator metrics.
2.3.2. Additional resources
2.4. Monitoring health information for application resources and deployments
The Red Hat OpenShift GitOps Environments page in the Developer perspective of the OpenShift Container Platform web console shows a list of the successful deployments of the application environments, along with links to the revision for each deployment.
The Application environments page in the Developer perspective of the OpenShift Container Platform web console displays the health status of the application resources, such as routes, synchronization status, deployment configuration, and deployment history.
The environments pages in the Developer perspective of the OpenShift Container Platform web console are decoupled from the Red Hat OpenShift GitOps Application Manager command-line interface (CLI), kam
. You do not have to use kam
to generate Application Environment manifests for the environments to show up in the Developer perspective of the OpenShift Container Platform web console. You can use your own manifests, but the environments must still be represented by namespaces. In addition, specific labels and annotations are still needed.
2.4.1. Settings for environment labels and annotations
This section provides reference settings for environment labels and annotations required to display an environment application in the Environments page, in the Developer perspective of the OpenShift Container Platform web console.
Environment labels
The environment application manifest must contain labels.openshift.gitops/environment
and destination.namespace
fields. You must set identical values for the <environment_name>
variable and the name of the environment application manifest.
Specification of the environment application manifest
spec: labels: openshift.gitops/environment: <environment_name> destination: namespace: <environment_name> # ...
Example of an environment application manifest
apiVersion: argoproj.io/v1beta1
kind: Application
metadata:
name: dev-env 1
namespace: openshift-gitops
spec:
labels:
openshift.gitops/environment: dev-env
destination:
namespace: dev-env
# ...
- 1
- The name of the environment application manifest. The value set is the same as the value of the
<environment_name>
variable.
Environment annotations
The environment namespace manifest must contain the annotations.app.openshift.io/vcs-uri
and annotations.app.openshift.io/vcs-ref
fields to specify the version controller code source of the application. You must set identical values for the <environment_name>
variable and the name of the environment namespace manifest.
Specification of the environment namespace manifest
apiVersion: v1
kind: Namespace
metadata:
annotations:
app.openshift.io/vcs-uri: <application_source_url>
app.openshift.io/vcs-ref: <branch_reference>
name: <environment_name> 1
# ...
- 1
- The name of the environment namespace manifest. The value set is the same as the value of the
<environment_name>
variable.
Example of an environment namespace manifest
apiVersion: v1 kind: Namespace metadata: annotations: app.openshift.io/vcs-uri: https://example.com/<your_domain>/<your_gitops.git> app.openshift.io/vcs-ref: main labels: argocd.argoproj.io/managed-by: openshift-gitops name: dev-env # ...
2.4.2. Checking health information
The Red Hat OpenShift GitOps Operator will install the GitOps backend service in the openshift-gitops
namespace.
Prerequisites
- The Red Hat OpenShift GitOps Operator is installed from OperatorHub.
- Ensure that your applications are synchronized by Argo CD.
Procedure
- Click Environments under the Developer perspective. The Environments page shows the list of applications along with their Environment status.
- Hover over the icons under the Environment status column to see the synchronization status of all the environments.
- Click the application name from the list to view the details of a specific application.
In the Application environments page, if the Resources section under the Overview tab displays icons, hover over the icons to get status details.
- A broken heart indicates that resource issues have degraded the application’s performance.
- A yellow yield sign indicates that resource issues have delayed data about the application’s health.
- To view the deployment history of an application, click the Deployment History tab. The page includes details such as the Last deployment, Description (commit message), Environment, Author, and Revision.
2.5. Monitoring Argo CD custom resource workloads
With Red Hat OpenShift GitOps, you can monitor the availability of Argo CD custom resource workloads for specific Argo CD instances. By monitoring Argo CD custom resource workloads, you have the latest information about the state of your Argo CD instances by enabling alerts for them. When the component workload pods such as application-controller, repo-server, or server of the corresponding Argo CD instance are unable to come up for certain reasons and there is a drift between the number of ready replicas and the number of desired replicas for a certain period of time, the Operator then triggers the alerts.
You can enable and disable the setting for monitoring Argo CD custom resource workloads.
2.5.1. Prerequisites
-
You have access to the cluster as a user with the
cluster-admin
role. - Red Hat OpenShift GitOps is installed in your cluster.
-
The monitoring stack is configured in your cluster in the
openshift-monitoring
project. In addition, the Argo CD instance is in a namespace that you can monitor through Prometheus. -
The
kube-state-metrics
service is running in your cluster. Optional: If you are enabling monitoring for an Argo CD instance already present in a user-defined project, ensure that the monitoring is enabled for user-defined projects in your cluster.
NoteIf you want to enable monitoring for an Argo CD instance in a namespace that is not watched by the default
openshift-monitoring
stack, for example, any namespace that does not start withopenshift-*
, then you must enable user workload monitoring in your cluster. This action enables the monitoring stack to pick up the created PrometheusRule.
2.5.2. Enabling Monitoring for Argo CD custom resource workloads
By default, the monitoring configuration for Argo CD custom resource workloads is set to false
.
With Red Hat OpenShift GitOps, you can enable workload monitoring for specific Argo CD instances. As a result, the Operator creates a PrometheusRule
object that contains alert rules for all the workloads managed by the specific Argo CD instances. These alert rules trigger the firing of an alert when the replica count of the corresponding component has drifted from the desired state for a certain amount of time. The Operator will not overwrite the changes made to the PrometheusRule
object by the users.
Procedure
Set the
.spec.monitoring.enabled
field value totrue
on a given Argo CD instance:Example Argo CD custom resource
apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: example-argocd labels: example: repo spec: # ... monitoring: enabled: true # ...
Verify whether an alert rule is included in the PrometheusRule created by the Operator:
Example alert rule
apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: argocd-component-status-alert namespace: openshift-gitops spec: groups: - name: ArgoCDComponentStatus rules: # ... - alert: ApplicationSetControllerNotReady 1 annotations: message: >- applicationSet controller deployment for Argo CD instance in namespace "default" is not running expr: >- kube_statefulset_status_replicas{statefulset="openshift-gitops-application-controller statefulset", namespace="openshift-gitops"} != kube_statefulset_status_replicas_ready{statefulset="openshift-gitops-application-controller statefulset", namespace="openshift-gitops"} for: 1m labels: severity: critical
- 1
- Alert rule in the PrometheusRule that checks whether the workloads created by the Argo CD instances are running as expected.
2.5.3. Disabling Monitoring for Argo CD custom resource workloads
You can disable workload monitoring for specific Argo CD instances. Disabling workload monitoring deletes the created PrometheusRule.
Procedure
Set the
.spec.monitoring.enabled
field value tofalse
on a given Argo CD instance:Example Argo CD custom resource
apiVersion: argoproj.io/v1beta1 kind: ArgoCD metadata: name: example-argocd labels: example: repo spec: # ... monitoring: enabled: false # ...