Chapter 11. Accessing metrics

Procedure

1. If you have not obtained the OpenShift Container Platform API route for Thanos Querier, run the following command:

   $ oc get routes -n openshift-monitoring thanos-querier -o jsonpath='{.status.ingress[0].host}'

2. Retrieve a list of metrics in JSON format from the Thanos Querier API route by running the following command. This command uses oc to authenticate with a bearer token.

   $ curl -k -H "Authorization: Bearer $(oc whoami -t)" https://<thanos_querier_route>/api/v1/metadata 1

   1

   Replace <thanos_querier_route> with the OpenShift Container Platform API route for Thanos Querier.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, click Observe and go to the Metrics tab.

2. To add one or more queries, perform any of the following actions:

   Option	Description
   Select an existing query.	From the Select query drop-down list, select an existing query.
   Create a custom query.	Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. Use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. Move your mouse pointer over a suggested item to view a brief description of that item.
   Add multiple queries.	Click Add query.
   Duplicate an existing query.	Click the options menu next to the query, then choose Duplicate query.
   Disable a query from being run.	Click the options menu next to the query and choose Disable query.

3. To run queries that you created, click Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

   Note
   • When drawing time series graphs, queries that operate on large amounts of data might time out or overload the browser. To avoid this, click Hide graph and calibrate your query by using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.
   • By default, the query table shows an expanded view that lists every metric and its current value. Click the ˅ down arrowhead to minimize the expanded view for a query.
4. Optional: Save the page URL to use this set of queries again in the future.

5. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. Select which metrics are shown by performing any of the following actions:

   Option	Description
   Hide all metrics from a query.	Click the options menu for the query and click Hide all series.
   Hide a specific metric.	Go to the query table and click the colored square near the metric name.
   Zoom into the plot and change the time range.	Perform one of the following actions: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu to select the time range.
   Reset the time range.	Click Reset zoom.
   Display outputs for all queries at a specific point in time.	Hover over the plot at the point you are interested in. The query outputs appear in a pop-up box.
   Hide the plot.	Click Hide graph.

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, go to Observe Targets. The Metrics targets page opens with a list of all service endpoint targets that are being scraped for metrics.

   This page shows details about targets for default OpenShift Container Platform and user-defined projects. This page lists the following information for each target:

   • Service endpoint URL being scraped
   • The ServiceMonitor resource being monitored
   • The up or down status of the target
   • Namespace
   • Last scrape time
   • Duration of the last scrape

2. Optional: To find a specific target, perform any of the following actions:

   Option	Description
   Filter the targets by status and source.	Choose filters in the Filter list. The following filtering options are available: Status filters: Up. The target is currently up and being actively scraped for metrics. Down. The target is currently down and not being scraped for metrics. Source filters: Platform. Platform-level targets relate only to default Red Hat OpenShift Service on AWS projects. These projects provide core Red Hat OpenShift Service on AWS functionality. User. User targets relate to user-defined projects. These projects are user-created and can be customized.
   Search for a target by name or label.	Enter a search term in the Text or Label field next to the search box.
   Sort the targets.	Click one or more of the Endpoint Status, Namespace, Last Scrape, and Scrape Duration column headers.

3. Click the URL in the Endpoint column for a target to go to its Target details page. This page provides information about the target, including the following information:

   • The endpoint URL being scraped for metrics
   • The current Up or Down status of the target
   • A link to the namespace
   • A link to the ServiceMonitor resource details
   • Labels attached to the target
   • The most recent time that the target was scraped for metrics

Procedure

1. In the Administrator perspective of the OpenShift Container Platform web console, go to Observe Dashboards.
2. Choose a dashboard in the Dashboard list. Some dashboards, such as etcd and Prometheus dashboards, produce additional sub-menus when selected.

3. Optional: Select a time range for the graphs in the Time Range list.

   • Select a pre-defined time period.

   • Set a custom time range by clicking Custom time range in the Time Range list.

     1. Input or select the From and To dates and times.
     2. Click Save to save the custom time range.
4. Optional: Select a Refresh Interval.
5. Hover over each of the graphs within a dashboard to display detailed information about specific items.

Procedure

1. If you have not obtained the OpenShift Container Platform API route for Thanos Querier, run the following command:

   $ oc get routes -n openshift-monitoring thanos-querier -o jsonpath='{.status.ingress[0].host}'

2. Retrieve a list of metrics in JSON format from the Thanos Querier API route by running the following command. This command uses oc to authenticate with a bearer token.

   $ curl -k -H "Authorization: Bearer $(oc whoami -t)" https://<thanos_querier_route>/api/v1/metadata 1

   1

   Replace <thanos_querier_route> with the OpenShift Container Platform API route for Thanos Querier.

Procedure

1. In the Developer perspective of the OpenShift Container Platform web console, click Observe and go to the Metrics tab.
2. Select the project that you want to view metrics for from the Project: list.

3. To add one or more queries, perform any of the following actions:

   Option	Description
   Select an existing query.	From the Select query drop-down list, select an existing query.
   Create a custom query.	Add your Prometheus Query Language (PromQL) query to the Expression field. As you type a PromQL expression, autocomplete suggestions appear in a drop-down list. These suggestions include functions, metrics, labels, and time tokens. Use the keyboard arrows to select one of these suggested items and then press Enter to add the item to your expression. Move your mouse pointer over a suggested item to view a brief description of that item.
   Add multiple queries.	Click Add query.
   Duplicate an existing query.	Click the options menu next to the query, then choose Duplicate query.
   Disable a query from being run.	Click the options menu next to the query and choose Disable query.

4. To run queries that you created, click Run queries. The metrics from the queries are visualized on the plot. If a query is invalid, the UI shows an error message.

   Note
   • When drawing time series graphs, queries that operate on large amounts of data might time out or overload the browser. To avoid this, click Hide graph and calibrate your query by using only the metrics table. Then, after finding a feasible query, enable the plot to draw the graphs.
   • By default, the query table shows an expanded view that lists every metric and its current value. Click the ˅ down arrowhead to minimize the expanded view for a query.
5. Optional: Save the page URL to use this set of queries again in the future.

6. Explore the visualized metrics. Initially, all metrics from all enabled queries are shown on the plot. Select which metrics are shown by performing any of the following actions:

   Option	Description
   Hide all metrics from a query.	Click the options menu for the query and click Hide all series.
   Hide a specific metric.	Go to the query table and click the colored square near the metric name.
   Zoom into the plot and change the time range.	Perform one of the following actions: Visually select the time range by clicking and dragging on the plot horizontally. Use the menu to select the time range.
   Reset the time range.	Click Reset zoom.
   Display outputs for all queries at a specific point in time.	Hover over the plot at the point you are interested in. The query outputs appear in a pop-up box.
   Hide the plot.	Click Hide graph.

Procedure

1. In the Developer perspective in the OpenShift Container Platform web console, click Observe and go to the Dashboards tab.
2. Select a project from the Project: drop-down list.

3. Select a dashboard from the Dashboard drop-down list to see the filtered metrics.

   Note

   All dashboards produce additional sub-menus when selected, except Kubernetes / Compute Resources / Namespace (Pods).

4. Optional: Select a time range for the graphs in the Time Range list.

   • Select a pre-defined time period.

   • Set a custom time range by clicking Custom time range in the Time Range list.

     1. Input or select the From and To dates and times.
     2. Click Save to save the custom time range.
5. Optional: Select a Refresh Interval.
6. Hover over each of the graphs within a dashboard to display detailed information about specific items.

Procedure

1. Extract an authentication token by running the following command:

   $ TOKEN=$(oc whoami -t)

2. Extract the alertmanager-main API route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route alertmanager-main -ojsonpath={.status.ingress[].host})

3. Query the service API receivers for Alertmanager by running the following command:

   $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v2/receivers"

Procedure

1. Retrieve the bearer token by running the following the command:

   $ TOKEN=$(oc whoami -t)

2. Get the Prometheus federation route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route prometheus-k8s-federate -ojsonpath={.status.ingress[].host})

3. Query metrics from the /federate route. The following example command queries up metrics:

   $ curl -G -k -H "Authorization: Bearer $TOKEN" https://$HOST/federate --data-urlencode 'match[]=up'

   Example output

   # TYPE up untyped
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.143.148:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035322214
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.148.166:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035338597
   up{apiserver="kube-apiserver",endpoint="https",instance="10.0.173.16:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035343834
   ...

Procedure

1. Extract an authentication token to connect to Prometheus by running the following command:

   $ TOKEN=$(oc whoami -t)

2. Extract the thanos-querier API route URL by running the following command:

   $ HOST=$(oc -n openshift-monitoring get route thanos-querier -ojsonpath={.status.ingress[].host})

3. Set the namespace to the namespace in which your service is running by using the following command:

   $ NAMESPACE=ns1

4. Query the metrics of your own services in the command line by running the following command:

   $ curl -H "Authorization: Bearer $TOKEN" -k "https://$HOST/api/v1/query?" --data-urlencode "query=up{namespace='$NAMESPACE'}"

   The output shows the status for each application pod that Prometheus is scraping:

   The formatted example output

   {
     "status": "success",
     "data": {
       "resultType": "vector",
       "result": [
         {
           "metric": {
             "__name__": "up",
             "endpoint": "web",
             "instance": "10.129.0.46:8080",
             "job": "prometheus-example-app",
             "namespace": "ns1",
             "pod": "prometheus-example-app-68d47c4fb6-jztp2",
             "service": "prometheus-example-app"
           },
           "value": [
             1591881154.748,
             "1"
           ]
         }
       ],
     }
   }

   Note
   • The formatted example output uses a filtering tool, such as jq, to provide the formatted indented JSON. See the jq Manual (jq documentation) for more information about using jq.
   • The command requests an instant query endpoint of the Thanos Querier service, which evaluates selectors at one point in time.