Chapter 3. Using observability

Get the default CA certificate and store the content of the key tls.crt in a local file. Run the following command:

oc -n openshift-ingress get secret router-certs-default -o jsonpath="{.data.tls\.crt}" | base64 -d > ca.crt

Run the following command to query metrics:

curl --cacert ./ca.crt -H "Authorization: Bearer {TOKEN}" https://{PROXY_ROUTE_URL}/api/v1/query?query={QUERY_EXPRESSION}

Note: The QUERY_EXPRESSION is the standard Prometheus query expression. For example, query the metrics cluster_infrastructure_provider by replacing the URL in the previously mentioned command with the following URL: https://{PROXY_ROUTE_URL}/api/v1/query?query=cluster_infrastructure_provider. For more details, see Querying Prometheus.

Run the following command to create proxy-byo-ca secrets using the generated certificates:

oc -n open-cluster-management-observability create secret tls proxy-byo-ca --cert ./ca.crt --key ./ca.key

Create proxy-byo-cert secrets using the generated certificates by using the following command:

oc -n open-cluster-management-observability create secret tls proxy-byo-cert --cert ./ingress.crt --key ./ingress.key

Create the Kubernetes secret for an external endpoint with the access information of the external endpoint in the open-cluster-management-observability namespace. View the following example secret:

apiVersion: v1
kind: Secret
metadata:
  name: victoriametrics
  namespace: open-cluster-management-observability
type: Opaque
stringData:
  ep.yaml: |
    url: http://victoriametrics:8428/api/v1/write
    http_client_config:
      basic_auth:
        username: test
        password: test

The ep.yaml is the key of the content and is used in the MultiClusterObservability custom resource in next step. Currently, observability supports exporting metrics to endpoints without any security checks, with basic authentication or with tls enablement. View the following tables for a full list of supported parameters:

HttpClientConfig

BasicAuth

TLSConfig

Add the writeStorage parameter to the MultiClusterObservability custom resource for adding a list of external endppoints that you want to export. View the following example:

spec:
  storageConfig:
    writeStorage: 1
    - key: ep.yaml
      name: victoriametrics

View the status of metric export after the metrics export is enabled by checking the acm_remote_write_requests_total metric.